ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understanding
Open accessAbstract
We introduce ZeroSCROLLS, a zero-shot benchmark for natural language understanding over long texts, which contains only test and small validation sets, without training data. We adapt six tasks from the SCROLLS benchmark, and add four new datasets, including two novel information fusing tasks, such as aggregating the percentage of positive reviews. Using ZeroSCROLLS, we conduct a comprehensive evaluation of both open-source and closed large language models, finding that Claude outperforms ChatGPT, and that GPT-4 achieves the highest average score. However, there is still room for improvement on multiple open challenges in ZeroSCROLLS, such as aggregation tasks, where models struggle to pass the naive baseline. As the state of the art is a moving target, we invite researchers to evaluate their ideas on the live ZeroSCROLLS leaderboard.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Many long-context benchmarks require models to follow instructions and only support evaluating instruction-tuned models.
“Many long-context benchmarks require models to follow instructions and only support evaluating instruction-tuned models (Zhang et al., 2024b; Shaham et al., 2023).”
From HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughly (2024) · cites this paper as Shaham et al., 2023
Section 3.2 (page unavailable) says ZeroSCROLLS gives each task an instruction specifying the task and output format, supporting the instruction-following part of the claim. The Open Source Models and §4.1 passages describe models evaluated, but do not say the benchmark only supports evaluating instruction-tuned models. The claimed restriction is therefore unsupported. (LLM, conf=0.84)
Experimental — accuracy not yet measuredExisting benchmarks mostly use zero-shot prompting, which leads to inconsistent output formats, especially for base models.
“Existing benchmarks mostly use zero-shot prompting (Shaham et al., 2023; Zhang et al., 2024b), which leads to inconsistent output formats, especially for base models.”
From HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughly (2024) · cites this paper as Shaham et al., 2023
Section 3.2 describes ZeroSCROLLS as a zero-shot benchmark, and Section 5 reports output-format discrepancies across models and tasks. The paper does not establish that most existing benchmarks use zero-shot prompting or that format problems are especially pronounced for base models. The cited passages have no page numbers in our copy. (LLM, conf=0.86)
Experimental — accuracy not yet measuredMost natural language datasets in existing benchmarks are too short to effectively test frontier long-context abilities (usually ≥128K).
“Inadequate lengths : Most natural language datasets in existing benchmarks (Shaham et al., 2023; An et al., 2024; Table 4) are too short to effectively test frontier long-context abilities (usually ≥ 128K).”
From HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughly (2024) · cites this paper as Shaham et al., 2023
The Introduction (page not available in our copy) says that zero-shot benchmarks such as HELM and BigBench mostly focus on short sequences, giving BigBench’s average of 77 words per input. This supports the concern about short benchmark inputs, but the paper does not establish that most natural-language datasets in existing benchmarks are too short to test abilities at ≥128K tokens. (LLM, conf=0.88)
Experimental — accuracy not yet measuredBAMBOO employs four metrics: accuracy for QA and language modeling, concordance index for sorting, pass@1 for code completion, and precision/recall/F1 for hallucination detection.
“As shown in Table 2, we employ four metrics to evaluate different tasks in our benchmark, i.e., accuracy for question answering and language modeling, concordance index (Shaham et al., 2023) for sorting, pass@1 (Chen et al., 2021) for code completion, and precision, recall, and F1 for hallucination detection.”
From BAMBOO: A Comprehensive Benchmark for Evaluating Long Text Modeling Capacities of Large Language Models (2023) · cites this paper as Shaham et al., 2023
The synthetic input in RULER reduces reliance on parametric knowledge, which interferes with the utilization of long-context input in realistic tasks.
“The synthetic input in RULER reduces reliance on parametric knowledge, which interferes with the utilization of long-context input in realistic tasks (Shaham et al., 2023; Bai et al., 2023).”
From RULER: What's the Real Context Size of Your Long-Context Language Models? (2024) · cites this paper as Shaham et al., 2023
Section 4.3 discusses how model size and input length affect performance on ZeroSCROLLS, but does not address interference from parametric knowledge. Searches for “parametric knowledge,” “synthetic input,” and knowledge-based shortcuts found no passage in our copy supporting the claim about RULER. (LLM, conf=0.76)
Experimental — accuracy not yet measured
Previous benchmarks largely focus on evaluating instruction-tuned models.
“Previous benchmarks largely focus on evaluating instruction-tuned models (Shaham et al., 2023; Zhang et al., 2024b).”
From HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughly (2024) · cites this paper as Shaham et al., 2023
The Introduction (page unavailable) says earlier zero-shot LLM benchmarks such as HELM and BigBench mostly focus on short sequences; it does not say they largely evaluate instruction-tuned models. The Open Source Models section identifies instruction-tuned models evaluated by ZeroSCROLLS itself. Searches for previous benchmarks and instruction-tuned models found no passage supporting the attributed claim. (LLM, conf=0.72)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
No references could be resolved yet.
Research works printed but not linked: 25
- Stefanos Angelidis, Reinald Kim Amplayo, Yoshihiko Suhara, Xiaolan Wang, and Mirella Lapata. 2021. Extractive opinion summarization in quantized trans- former spaces. Transactions of the Association for Computational Linguistics, 9:277-293. · DOI · Paper page
- Mingda Chen, Zewei Chu, Sam Wiseman, and Kevin Gimpel. 2022. SummScreen: A dataset for abstrac- tive screenplay summarization. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 8602-8615, Dublin, Ireland. Association for Compu- tational Linguistics. · DOI · Paper page
- Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A. Smith, and Matt Gardner. 2021. A dataset of information-seeking questions and answers an- chored in research papers. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, pages 4599-4610, On- line. Association for Computational Linguistics. · DOI · Paper page
- Mandy Guo, Joshua Ainslie, David Uthus, Santiago On- tanon, Jianmo Ni, Yun-Hsuan Sung, and Yinfei Yang. 2022. LongT5: Efficient text-to-text transformer for long sequences. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 724- 736, Seattle, United States. Association for Compu- tational Linguistics. · DOI · Paper page
- Luyang Huang, Shuyang Cao, Nikolaus Parulian, Heng Ji, and Lu Wang. 2021. Efficient attentions for long document summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1419-1436, Online. Association for Computational Linguistics. · DOI · Paper page
- Maor Ivgi, Uri Shaham, and Jonathan Berant. 2023. Efficient Long-Text Understanding with Short-Text Models. Transactions of the Association for Compu- tational Linguistics, 11:284-299. · DOI · Paper page
- Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Ed- ward Grefenstette. 2018. The NarrativeQA reading comprehension challenge. Transactions of the Asso- ciation for Computational Linguistics, 6:317-328. · DOI · Paper page
- Wojciech Kryscinski, Nazneen Rajani, Divyansh Agar- wal, Caiming Xiong, and Dragomir Radev. 2022. BOOKSUM: A collection of datasets for long-form narrative summarization. In Findings of the Associ- ation for Computational Linguistics: EMNLP 2022, pages 6536-6558, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
- Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045-3059, Online and Punta Cana, Domini- can Republic. Association for Computational Lin- guistics. · DOI · Paper page
- Chin-Yew Lin. 2004. ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- tion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
- Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kin- ney, and Daniel Weld. 2020. S2ORC: The semantic scholar open research corpus. In Proceedings of the 58th Annual Meeting of the Association for Compu- tational Linguistics, pages 4969-4983, Online. Asso- ciation for Computational Linguistics. OpenAI. 2023. Gpt-4 technical report. · DOI · Paper page
- Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems.
- Richard Yuanzhe Pang, Alicia Parrish, Nitish Joshi, Nikita Nangia, Jason Phang, Angelica Chen, Vishakh Padmakumar, Johnny Ma, Jana Thompson, He He, and Samuel Bowman. 2022. QuALITY: Question answering with long input texts, yes! In Proceedings of the 2022 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 5336-5358, Seattle, United States. Association for Computational Linguistics. · DOI · Paper page
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
- Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natu- ral Language Processing, pages 2383-2392, Austin, Texas. Association for Computational Linguistics. · DOI · Paper page
- Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal V. Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Tr- ishala Neeraj, Jos Rozen, Abheesht Sharma, An- drea Santilli, Thibault Févry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M. Rush. 2022. Multi- task prompted training enables zero-shot task gener- alization. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
- Timo Schick and Hinrich Schütze. 2021a. Exploiting cloze-questions for few-shot text classification and natural language inference. In Proceedings of the 16th Conference of the European Chapter of the Asso- ciation for Computational Linguistics: Main Volume, pages 255-269, Online. Association for Computa- tional Linguistics. · DOI · Paper page
- Timo Schick and Hinrich Schütze. 2021b. Few-shot text generation with natural language instructions. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 390- 402, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. · DOI · Paper page
- Uri Shaham, Elad Segal, Maor Ivgi, Avia Efrat, Ori Yoran, Adi Haviv, Ankit Gupta, Wenhan Xiong, Mor Geva, Jonathan Berant, and Omer Levy. 2022. SCROLLS: Standardized CompaRison over long lan- guage sequences. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 12007-12021, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
- Yi Tay, Mostafa Dehghani, Vinh Q. Tran, Xavier Garcia, Jason Wei, Xuezhi Wang, Hyung Won Chung, Dara Bahri, Tal Schuster, Steven Zheng, Denny Zhou, Neil Houlsby, and Donald Metzler. 2023. UL2: Unifying language learning paradigms. In The Eleventh Inter- national Conference on Learning Representations.
- Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022. MuSiQue: Multi- hop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539-554. · DOI · Paper page
- Alex Wang, Richard Yuanzhe Pang, Angelica Chen, Ja- son Phang, and Samuel R. Bowman. 2022. SQuAL- ITY: Building a long-document summarization dataset the hard way. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 1139-1156, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
- Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, An- drew M. Dai, and Quoc V. Le. 2022a. Finetuned language models are zero-shot learners. In The Tenth International Conference on Learning Representa- tions, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
- Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022b. Chain-of-thought prompt- ing elicits reasoning in large language models. In Advances in Neural Information Processing Systems, volume 35, pages 24824-24837.
- Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan Awadallah, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, and Dragomir Radev. 2021. QMSum: A new benchmark for query- based multi-domain meeting summarization. In Pro- ceedings of the 2021 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5905-5921, Online. Association for Computational Linguistics. · DOI · Paper page
Cited by
Sorted by citations
Lost in the Middle: How Language Models Use Long Contexts
Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00638 · Cited by 1,612 · see citing works
Levy, M., Jacoby, A., & Goldberg, Y. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.818 · Cited by 119 · see citing works
A Survey on the Memory Mechanism of Large Language Model-based Agents
Zhang, Z., Dai, Q., Bo, X., Ma, C., Li, R., Chen, X., Zhu, J., Dong, Z., & Wen, J.-R. (2025) · ACM Transactions on Information Systems · doi:10.1145/3748302 · Cited by 118 · see citing works
LooGLE: Can Long-Context Language Models Understand Long Contexts?
Li, J., Wang, M., Zheng, Z., & Zhang, M. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.859 · Cited by 55 · see citing works
RULER: What's the Real Context Size of Your Long-Context Language Models?
Hsieh, C.-P., Sun, S., Kriman, S., Acharya, S., Rekesh, D., Jia, F., Zhang, Y., & Ginsburg, B. (2024) · arXiv · Cited by 13 · see citing works
LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression
Jiang, H., Wu, Q., Luo, X., Li, D., Lin, C.-Y., Yang, Y., & Qiu, L. (2023) · arXiv · Cited by 12 · see citing works
“(5) We evaluate LongLLMLingua across five benchmarks, i.e. , Nat-uralQuestions (Liu et al., 2024), LongBench (Bai et al., 2023), ZeroSCROLLS (Shaham et al., 2023), MuSicQue (Trivedi et al., 2022), and LooGLE (Li et al., 2023b), covering a variety of long context scenarios.”
MethodInfluentialA Survey on the Memory Mechanism of Large Language Model based Agents
Zhang, Z., Bo, X., Ma, C., Li, R., Chen, X., Dai, Q., Zhu, J., Dong, Z., & Wen, J.-R. (2024) · arXiv · doi:10.48550/arxiv.2404.13501 · Cited by 149 · see citing works
L-Eval: Instituting Standardized Evaluation for Long Context Language Models
An, C., Gong, S., Zhong, M., Zhao, X., Li, M., Zhang, J., Kong, L., & Qiu, X. (2023) · arXiv · doi:10.48550/arxiv.2307.11088 · Cited by 6 · see citing works
BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack
Kuratov, Y., Bulatov, A., Anokhin, P., Rodkin, I., Sorokin, D., Sorokin, A., & Burtsev, M. (2024) · arXiv · doi:10.48550/arxiv.2406.10149 · Cited by 19 · see citing works
ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference
Liu, X., Tang, Z., Dong, P., Li, Z., Liu, Y., Li, B., Hu, X., & Chu, X. (2025) · arXiv · doi:10.48550/arxiv.2502.00299 · Cited by 4 · see citing works
“Complementing these efforts, ZeroSCROLLS [68] and L-Eval [69] have broadened the scope to include a diverse array of practical natural language tasks, including query-driven summarization.”
Dong, Z., Tang, T., Li, J., Zhao, W. X., & Wen, J.-R. (2023) · arXiv · doi:10.48550/arxiv.2309.13345 · Cited by 7 · see citing works
“Recently, with LLM drawing a great deal of attention, Ze-roSCROLLS (Shaham et al., 2023), L-Eval (An et al., 2023), and LongBench (Bai et al., 2023) were released, focusing on the zero-shot long text comprehension ability of LLMs.”
BackgroundLLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression
Pan, Z., Wu, Q., Jiang, H., Xia, M., Luo, X., Zhang, J., Lin, Q., Rühle, V., Yang, Y., Lin, C.-Y., et al. (2024) · arXiv · Cited by 2 · see citing works
“Firstly, we have conducted extensive out-of-domain evaluation on four benchmarks in the paper, including LongBench (Bai et al., 2023), Zero-SCROLLS (Shaham et al., 2023), GSM8K (Cobbe et al., 2021), and Big Bench Hard (BBH) (bench authors, 2023), which cover multiple tasks from document QA to math problems and in-cont…”
MethodInfluentialHELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughly
Yen, H., Gao, T., Hou, M., Ding, K., Fleischer, D., Izsak, P., Wasserblat, M., & Chen, D. (2024) · arXiv · Cited by 2 · see citing works
“As discussed in the main paper, existing benchmarks either are limit to relatively short context lengths (Shaham et al., 2022; 2023; Li et al., 2024a; An et al., 2024; Bai et al., 2024; Dong et al., 2024) or lack rigorous evaluation methods (Zhang et al., 2024; Yuan et al., 2024).”
BackgroundMethodInfluentialA Multi-Agent Framework for Accessible Web Code Generation
Aljedaani, W., Shaik, M. A., & Aljohani, A. (2026) · Proceedings of the 23rd International Web for All Conference · doi:10.1145/3800424.3800453 · Cited by 1 · see citing works
Cao, S., Zou, K., & Wang, L. (2025) · Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2025.emnlp-main.1707
“Early benchmarks for long-context evaluation primarily focus on realistic tasks, where data is either sourced from human-annotated documents (Pang et al., 2022) or existing text corpora (Shaham et al., 2023; Dong et al., 2024).”
Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.21783 · Cited by 505 · see citing works
- Authors
- Shaham, U., Ivgi, M., Efrat, A., Berant, J., & Levy, O.
- Affiliations
- Tel Aviv University (IL)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- Pages
- 7977-7989
- arXiv
- 2305.14196
- OpenAlex
- W4389519019
- Cited by
- 23
- References
- 0
- Citations by year
- 2026: 9 · 2025: 7 · 2024: 6 · 2023: 1
- FWCI
- 4.07
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- Tel Aviv University
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- zero-shot learning, natural language understanding, benchmark, data fusion, large language models, model evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
