CoChat
Sign Up
PreprintCited by 27Topic ModelingNatural Language Processing TechniquesInformation Retrieval and Search Behavior

L-Eval: Instituting Standardized Evaluation for Long Context Language Models

Open access
An, Chenxin, Gong, Shansan, Zhong, Ming, Zhao, Xingjian, Li, Mukai, Zhang, Jun, Kong, Lingpeng, Qiu, Xipeng·2023·arXiv
DOI 10.48550/arxiv.2307.11088arXiv 2307.11088
Open this paper in CoChat — free accountOpen PDF

Abstract

Recently, there has been growing interest in extending the context length of large language models (LLMs), aiming to effectively process long inputs of one turn or conversations with more extensive histories. While proprietary models such as GPT-4 and Claude can largely preserve the reasoning ability in an extended context, open-source models are still progressing through the early stages of development. To bridge this gap, we propose L-Eval to institute a more standardized evaluation for long context language models (LCLMs) addressing two key aspects: dataset construction and evaluation metrics. On the one hand, we build a new evaluation suite containing 20 sub-tasks, 508 long documents, and over 2,000 human-labeled query-response pairs encompassing diverse question styles, domains, and input length (3k$\sim$200k tokens). On the other hand, we investigate the effectiveness in evalution metrics for LCLMs. Results show that popular n-gram matching metrics generally can not correlate well with human judgment, and thus we strongly advocate for length-instruction-enhanced (LIE) evaluation and employing LLM judges. We conducted a comprehensive study of 4 popular commercial LLMs and 12 open-source counterparts using the L-Eval benchmark. Our empirical findings offer useful insights into the study of LCLMs and lay the groundwork for the development of more principled evaluation of these models.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2307.11088

Cites

0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

5 of 14 cited sources checked · 3 supported · 0 flagged · accuracy not yet measured

  • Findingown data

    LLM judges provide superior accuracy in the evaluation of open-ended tasks.

    “Our further experiments suggest that LLM judges (Li et al., 2023c;Zheng et al., 2023) provide superior accuracy in the evaluation of open-ended tasks.”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites Li et al., 2023c

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Zheng et al., 2023 source

      Section 3 (p. 4) says conventional reference-based metrics are ineffective for the paper’s open-ended questions, and §4.2 (p. 7) reports that GPT-4 judgments closely agree with human preferences. This supports the usefulness of an LLM judge for those tasks, but the paper does not establish superior accuracy for LLM judges generally across open-ended tasks; its strongest result is specifically for GPT-4 on MT-bench and Chatbot Arena. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
  • Premisecitation

    LLM evaluators favor more detailed and lengthy answers.

    “LLM evaluators have been reported to favor more detailed and lengthy answers (Zheng et al., 2023).”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Zheng et al., 2023 source
  • Findingown data

    The performance of NTK variants is strongly affected by the maximum tokens of the dataset.

    “We can observe that employing which variants of NTK are strongly affected by the maximum tokens of the dataset.”
    Quote in the paperSubstantial claim
  • Findingown data

    LIE evaluation substantially improves the Kendall-Tau correlation coefficient with human judgments for all automatic metrics.

    “The empirical results demonstrate a substantial improvement brought by LIE evaluation in the Kendall-Tau correlation coefficient (τ ) with human judgments (Figure 2), for all automatic metrics.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Human evaluation is the most accurate evaluation for open-ended tasks.
    • Rests on an assumption: Human scores from the three annotators are a reliable ground truth for evaluating automatic metrics.
  • Findingown data

    Increasing the base from 20,000 to 160,000 continuously improves topic retrieval.

    “when we increase the base from 20,000 to 160,000, there is a continuous improvement on topic retrieval.”
    Quote in the paperSubstantial claim
  • Classified as probably not a substantial claim. 28
  • May combine two claims. 28
  • The quote only partly states this claim. 1

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    MultiDoc2Dial aims to model goal-oriented dialogues grounded in multiple documents.

    “This dataset is sampled from the MultiDoc2Dial dataset (Feng et al., 2021) which aims to model goal-oriented dialogues grounded in multiple documents.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Feng et al., 2021 source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    SummScreen is an abstractive summarization dataset combining TV series transcripts and episode recaps, constructed from fan-contributed websites.

    “This dataset originates from the SummScreen (Chen et al., 2022), the original dataset is an abstractive summarization dataset combining TV series transcripts and episode recaps.SummScreen is constructed from fan-contributed websites.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Chen et al., 2022 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    LLMs have acquired many commonsense in their pretraining corpus known as parametric knowledge.

    “LLMs have acquired many commonsense in their pertaining corpus known as parametric knowledge (Wang et al., 2023).”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Wang et al., 2023 source

      The Introduction (page not available in our copy) says LLMs hold parametric knowledge and links their information to pre-training data. It does not specifically say that they acquired commonsense knowledge; “commonsense” appears only in a reference title. (LLM, conf=0.82)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Findingreasoning

    Increased input length can lower the model's ability to comprehend lengthy instructions, thereby inhibiting its capability to generate answers that closely match the length of the ground truth.

    “As we mentioned before, the increased input length can also lower the model's ability to comprehend lengthy instructions, thereby inhibiting its capability to generate answers that closely match the length of the ground truth.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Fine-tuning longer offers benefits for closed-ended tasks but falls short in open-ended tasks.

    “Fine-tuning longer offers benefits for closed-ended tasks but falls short in open-ended tasks”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    In the authors' experiments, the NTK-aware positional embedding scaling theory does not hold in practical tasks, and the model still tends to generate random tokens when setting t = 4 on 16k context length.

    “In our experiments, this theory does not hold in practical tasks (see section §A.3), which means the model still tends to generate random tokens when setting t = 4 on 16k context length.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    GPT-4-32k is more capable of closed-ended tasks, demonstrating powerful reasoning ability over long context, while the 100k context length helps Claude surpass both GPT-4-32k and Turbo-16k on open-ended tasks.

    “We find that GPT-4-32k is more capable of closed-ended tasks demonstrating powerful reasoning ability over long context since most closed-ended task in L-Eval has less than 32k input tokens, but the 100k context length help Cluade surpass both GPT-4-32k and Turbo-16k on open-ended tasks which generally has more input tokens.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    In closed-ended tasks, bm25 often matches the performance of the dense retriever ada-embedding.

    “evaluating retrievers, bm25 often matches the performance of the dense retriever, ada-embedding, in closed-ended tasks.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    CodeU and GSM(16-shot) in L-Eval cannot be solved by retrieval.

    “Our results show that CodeU and GSM(16-shot) in L-Eval can not be solved by retrieval.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The maximum sample in L-Eval contains nearly 200k tokens.

    “The maximum sample in L-Eval contains nearly 200k tokens.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The average input length in L-Eval ranges from 4k to 60k tokens.

    “The average input length in L-Eval ranges from 4k to 60k.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning

    Current efforts on open-source LCLMs improve performance on closed-ended tasks but significantly fall short on open-ended tasks.

    “(2) While current efforts on open-source LCLMs improve performance on closed-ended tasks, they significantly fall short on open-ended tasks.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Coursera is the first multi-choice dataset with multiple correct answers and is more challenging than single-option questions.

    “To the best of our knowledge, this is the first multi-choice dataset with multiple correct answers and it is more challenging than single-option questions (Table 3).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Turbo-16k struggles on SFcition and tends to answer questions relying on parametric knowledge.

    “We find that Turbo-16k struggles on this task, which tends to answer questions relying on parametric knowledge (Table 3).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    LLM judges are more accurate and robust to output length compared with N-gram metrics.

    “Compared with N-gram metrics, LLM judges are more accurate and robust to output length.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Human evaluation is the most accurate evaluation for open-ended tasks.
    • Rests on an assumption: Human scores from the three annotators are a reliable ground truth for evaluating automatic metrics.

  • May combine two claims.

    Findingown data

    Length-Instruction-Enhanced (LIE) evaluation is simple but effective in overcoming the length bias.

    “We first adopt Length-Instruction-Enhanced (LIE) evaluation in LLMs evaluation benchmarks which is simple but effective in overcoming the length bias, i.e., the number of words of ground truth is directly exposure to LCLMs.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The annotation quality of previous long sequence datasets fluctuates severely, and many unanswerable questions are unrelated to the context.

    “we find the annotation quality of previous long sequence datasets fluctuates severely and there are many unanswerable questions that are unrelated to the context.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    Retrieval-based approaches would never answer questions like how many sentences there are in a document.

    “For example, they would never answer questions like how many sentences are there in a document.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    The phenomenon of increased input length lowering instruction comprehension is less likely to be observed with more sophisticated LLMs such as Turbo-16k.

    “This phenomenon is less likely to be observed with more sophisticated LLMs (i.e.Turbo-16k).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    After adding length instructions, Kendall-Tau correlation improved from 0.5 to 0.8 for ROUGE-L, and the Kendall-Tau correlation of the GPT-4 evaluator reached 1.

    “after adding the length instructions, τ has been improved from 0.5 to 0.8 for ROUGE-L and τ of GPT-4 evaluator has even reached to 1.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Human evaluation is the most accurate evaluation for open-ended tasks.
    • Rests on an assumption: Human scores from the three annotators are a reliable ground truth for evaluating automatic metrics.

  • May combine two claims.

    Findingown data

    L-Eval contains 20 sub-tasks, 508 long documents, and over 2,000 human-labeled query-response pairs spanning diverse question styles, domains, and input lengths from 3k to 200k tokens.

    “we build a new evaluation suite containing 20 sub-tasks, 508 long documents, and over 2,000 human-labeled query-response pairs encompassing diverse question styles, domains, and input length (3k∼200k tokens).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The best model Vicuna1.5-13b-16k only wins Turbo-16k by 34%, 8 points lower than its short version Llama2-13b.

    “the best model Vicuna1.5-13b-16konly wins Turbo-16k by 34%, 8 points lower than its short version Llama2-13b.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Most of the answers to the science fiction questions contradict real-world principles and do not comply with actual physical laws, such as the statement that humans have invented the time machine.

    “Most of the answers to these questions contradict real-world principles and do not comply with actual physical laws, such as the statement: Humans have invented the time machine.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    There is a significant gap between open-source LCLMs and commercial models for both closed-ended and open-ended tasks.

    “(1) There is still a significant gap between open-source LCLMs and commercial models, for both closed-ended tasks (Table 3) and open-ended tasks evaluated by LLMs and human (Table 4, 5).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    When the input length is between 4k and 8k, NTK+bias gets the best results, and the NTK-weighted baseline is more robust on 16k input tokens.

    “When the input length is between 4k and 8k, NTK+bias gets the best results and NTK-weighted baseline is more robust on 16k input tokens.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The number of invalid outputs from Turbo-16k remains very small on both PART-A and PART-B, while invalid outputs from Llama2-16k dramatically increase on samples with longer input.

    “Results show that the num- ber of invalid outputs from Turbo-16k remains a very small amount on both PART-A and B while the invalid outputs from Llama2-16k dramatically increase on samples with longer input.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    Retrieval-based methods face difficulties in automatically identifying the query from user inputs.

    “Retrieval-based methods also face difficulties in automatically identifying the query from user inputs.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    L-Eval has 20 sub-tasks: 4 annotated from scratch, 4 re-annotated from public datasets, and 12 manually cleaned from previous long sequence datasets.

    “L-Eval has 20 sub-tasks, 4 sub-tasks are annotated from scratch ( §3.1), 4 sub-tasks are re-annotated from the public datasets ( §3.2), and the remaining 12 sub-tasks are manually cleaned from previous long sequence datasets.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    The documents and summaries in the government report summarization dataset are longer compared to other long document summarization datasets.

    “The documents and summaries in this dataset are longer compared to other long document summarization datasets.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    Evaluating outputs from the 96-question subset with GPT-4 only needs about $5.

    “Evaluating outputs from the 96-question subset with GPT-4 only needs about $5.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    Testing the 4 datasets in Table 2 needs about $100.

    “Testing the 4 datasets in Table2needs about $100!”
    Quote in the paperNot a substantial claim

  • May combine two claims. The quote only partly states this claim.

    Interpretationreasoning

    The assumption that the LLM evaluator is a super model does not hold in long context settings because it is impossible to feed the entire lengthy inputs into LLMs like GPT-4.

    “These works assume the LLM evaluator is a”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    Each sample in the human evaluation was scored by three annotators, all of whom are Ph.D. students researching long context language models.

    “Each sample is scored by three annotators, all of whom are Ph.D. students researching long context language models.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    The shortfall of open-source LCLMs on open-ended tasks is largely due to the models' misunderstanding of instructions as the input context length increases.

    “This is largely due to the models' misunderstanding of instructions as the input context length increases.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    Almost all metrics, including the GPT-4 evaluator, exhibit bias.

    “As previously discussed, almost all metrics, including the highly accurate automatic metric, the GPT-4 evaluator, exhibit bias.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationnone given

    Results tested with N-gram metrics may not be accurate when the performance of the models is very similar or there is a large difference in the granularity of the output.

    “For results tested with N-gram metrics, please note that the results may not be accurate when the performance of the models is very similar or there is a large difference in the granularity of the output.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    To minimize variance, the authors used greedy search as the decoding algorithm.

    “To minimize variance, we use greedy search as the decoding algorithm.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    The similar or superior summarization results of retrieve-based models may be because some paragraphs resembling summaries can be retrieved.

    “This may be because some paragraphs resembling summaries can be retrieved.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    The main reason why regular Turbo-0613 outperforms Turbo-16k is its superior ability to accurately follow instructions.

    “Besides, we also noticed that the main reason why regular Turbo-0613 outperforms Turbo-16k is its superior ability to accurately follow instructions.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Open-source models have very serious hallucination problems on the science fiction dataset, which in turn help them acquire a high score on this dataset.

    “As a result, open-source models have very serious hallucination problems which in turn help them acquire a high score on this dataset.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductown data

    In L-Eval, diversity and quality were prioritized over quantity, and all samples were manually validated after data collection.

    “In the design of L-Eval, we prioritize diversity and quality over quantity, ensuring correctness by manually validating all samples after data collection ( §3.3).”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    Language models are good at retrieving information from the very beginning or end of their input context but are usually lost in the middle.

    “Recent studies show that language models are good at retrieving information from the very beginning or end of its input context but are usually lost in the middle (Liu et al., 2023).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Liu et al., 2023

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The government report summarization dataset consists of long reports written by U.S. government research agencies such as the Congressional Research Service and Government Accountability Office.

    “This dataset is filtered from the government report summarization dataset (Huang et al., 2021), the dataset consists of long reports written by U.S. government research agencies such as the Congressional Research Service and Government Accountability Office.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Huang et al., 2021 source

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationreasoning

    In LCLMs, contextual knowledge is more crucial than parametric knowledge.

    “However, we argue that in LCLMs, contextual knowledge is more crucial than parametric knowledge.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Many long documents in real-world applications are private and can never be seen during pretraining.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    QMSum contains query-based meeting summarizations.

    “This dataset sourced from the QMSum (Zhong et al., 2021), this dataset contains query-based meeting summarizations.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Zhong et al., 2021 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    BigPatent consists of 1.3 million records of U.S. patent documents along with human-written abstractive summaries.

    “This dataset is derived from the BigPatent (Sharma et al., 2019) project, which consists of 1.3 million records of U.S. patent documents along with human-written abstractive summaries”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Sharma et al., 2019 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The original task of SPACE asks the model to summarize hotels from multiple aspects: food, location, cleanliness, etc.

    “The original task asks the model to summarize hotels from multiple aspects: food, location, cleanliness, etc.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Angelidis et al., 2021 source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductreasoning

    The authors reduced positional biases by swapping paired predictions, so the GPT-4 evaluator was used in 96×2 evaluation rounds, while the GPT-3.5 evaluator was used in 181×2 rounds.

    “We reduce the positional biases by swapping paired predictions, so the GPT-4 evaluator is used in 96×2 evaluation rounds, while the GPT3.5 evaluator is used in 181×2 rounds”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    SPACE consists of customer reviews of hotels from TripAdvisor, with 1.1 million training reviews for 11,000 hotels.

    “SPACE consists of customer reviews of hotels from TripAdvisor, with 1.1 million training reviews for 11,000 hotels.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Angelidis et al., 2021 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    CUAD is designed for supporting NLP research for automating legal contract review.

    “Questions on the Legal domain are drawn from the CUAD (Contract Understanding Atticus Dataset) dataset (Hendrycks et al., 2021b) designed for supporting NLP research for automating legal contract review.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Hendrycks et al., 2021b

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

50 of 50

  • The paper relies on this without support.

    Human evaluation is the most accurate evaluation for open-ended tasks.

    “Human evaluation This is the most accurate evaluation for open-ended tasks.”

  • The paper relies on this without support.

    LLMs cannot replace human evaluation.

    “Despite that some works show GPT-4 can be coherent with human judgment, LLMs cannot replace human evaluation.”

  • The paper relies on this without support.

    The 85-question subset is representative of open-ended tasks for human evaluation.

    “To save human laboratories, we propose a subset used for the human evaluation which has 12 long documents with 85 open-ended questions (85-question subset).”

  • The paper relies on this without support.

    The 96-question subset is representative for GPT-4 evaluation.

    “Thus we manually split a subset for GPT-4 evaluation consisting of 17 diverse long documents with 96 open-ended questions (96-question subset).”

  • The paper relies on this without support.

    Human scores from the three annotators are a reliable ground truth for evaluating automatic metrics.

    “We have 3 annotators to verify 7 models and calculate the Kendall-Tau correlation coefficient (τ ) between these metrics and the average human score.”

  • The paper relies on this without support.

    Prompting the LLM judge not to treat details absent from the reference answer as beneficial mitigates its bias.

    “Therefore, the judgment model must bear in mind that details not corroborated by the reference answers should not be considered beneficial.”

  • The paper relies on this without support.

    Human evaluation is the most equitable metric to assess long-context language models.

    “Consequently, human evaluation may be the most equitable metric to assess these models.”

  • The paper relies on this without support.

    Testing short context models on the topic retrieval task with truncated input texts is unfair.

    “Testing short context models on this task with truncated input texts is unfair, so we only include long context LLMs.”

  • The paper relies on this without support.

    The language model should follow contextual knowledge instead of parametric knowledge.

    “The language model should follow contextual knowledge instead of parametric knowledge.”

  • The paper relies on this without support.

    Many long documents in real-world applications are private and can never be seen during pretraining.

    “In real-world applications, many long documents are private and can never be seen during pretraining.”
10 of 10

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Cost assessment for full GPT-4 evaluation reverses

    Minor

    One section says evaluating all 1,000-plus open-ended questions with GPT-4 is unaffordable; the appendix says testing all open-ended cases with GPT-4 is affordable.

  • Multi-News sample counts disagree

    Minor

    Table 1 lists 11 Multi-News documents and instructions, whereas the dataset description says 10 articles were selected.

  • QuALITY document counts disagree

    Minor

    Table 1 lists 15 QuALITY documents, while the dataset description says 20 long stories were selected.

  • Turbo-16k has conflicting input limits

    Minor

    Table 4 assigns Turbo-16k-0613 a 4k token limit, but Table 3 assigns the same model 16k. The limit matters to the comparison between full-context and retrieval-based methods.

  • Re-annotated task count changes from four to five

    Minor

    The dataset breakdown assigns four tasks to re-annotation, but the later description says five publicly available datasets were re-annotated.

  • LongFQA question counts disagree

    Minor

    Table 1 lists 52 instructions for LongFQA, but its dataset description reports 54 questions.

  • Chatglm2 query-task average does not match its scores

    Minor

    The nine task scores in this row average about 17.09, not the reported 18.36. Other rows in the same table use the mean of their nine task scores for this final column.

    Reported 18.36 · recomputed 17.092222

7 of 7

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 4

  1. [11]11. In S. Koyejo, S. Mo- hamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural In- formation Processing Systems, volume 35, pp. 16344-16359.
  2. [12]12. Curran Associates, Inc., 2022. URL https://proceedings.neurips.cc/paper_files/paper/2022/file/ 67d57c32e20fd0a7a302cb81d36e40d5-Paper-Conference.pdf.
  3. [29]29. Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74-81, Barcelona, Spain, July 2004. Association for Computational Linguis- tics. URL https://aclanthology.org/W04-1013.
  4. [31]31. LocalLLaMA. Dynamically scaled rope further increases performance of long context llama with zero fine-tuning, July 2023a. URL https://www.reddit.com/r/LocalLLaMA/comments/ 14mrgpr/dynamically_scaled_rope_further_increases/. preprint

Cited by

Sorted by citations

Related

Authors
An, C., Gong, S., Zhong, M., Zhao, X., Li, M., Zhang, J., Kong, L., & Qiu, X.
Affiliations
University of Illinois Urbana-Champaign (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2307.11088
arXiv
2307.11088
OpenAlex
W4385009323
Semantic Scholar
b0db25e317cf856f1ec1ca3df0e573d850ed4696
Cited by
6
References
0
Citations by year
2026: 1 · 2025: 3 · 2024: 2
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
Keywords
long-context language models, standardized assessment, evaluation metrics, n-gram matching, LLM-as-a-judge, open-source large language models

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-11.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-11.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review