SCROLLS: Standardized CompaRison Over Long Language Sequences
Open accessAbstract
Uri Shaham, Elad Segal, Maor Ivgi, Avia Efrat, Ori Yoran, Adi Haviv, Ankit Gupta, Wenhan Xiong, Mor Geva, Jonathan Berant, Omer Levy. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Specifically designed evaluation suites such as Scrolls and ZeroScrolls exhibit a limited range of task types and input lengths, with summarization and abstractive QA predominant and average input length less than 16k tokens.
“Specifically designed evaluation suites, such as Scrolls and Ze-roScrolls (Shaham et al., 2022(Shaham et al., , 2023) ) have been proposed.However, they exhibit a limited range of task types and input lengths; tasks like summarization and abstractive QA are overwhelmingly predominant and usually have an average input length of less than 16k tokens.”
From L-Eval: Instituting Standardized Evaluation for Long Context Language Models (2024) · cites this paper as Shaham et al., 2022
The Introduction identifies SCROLLS as a seven-dataset suite, and §3.3 lists three summarization and two question-answering datasets, supporting the claim that these task types predominate. The dataset overview reports average input lengths in words, not tokens, so it does not establish the stated 16k-token threshold. This 2022 paper also does not establish the claim about ZeroScrolls. (LLM, conf=0.86)
Experimental — accuracy not yet measuredEarly work characterized memory capabilities through evaluations of long-context and long-range dependencies, focusing on sequence understanding and information retention.
“Early work characterized memory capabilities through evaluations of long-context and long-range dependencies [Shaham et al., 2022], focusing on sequence understanding and information retention.”
From VehicleMemBench: An Executable Benchmark for Multi-User Long-Term Memory in In-Vehicle Agents (2026) · cites this paper as Shaham et al., 2022
The Introduction (page unavailable) describes SCROLLS as an evaluation of long-text tasks in which critical information is spread across distant parts of the input. That supports the long-context and sequence-understanding aspect of the claim. The paper does not explicitly characterize its evaluations as measuring memory capabilities or information retention. (LLM, conf=0.82)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821 · see citing works
Self-citationTransformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.6 · Cited by 8,741 · see citing works
Natural Questions: A benchmark for question answering research
Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., et al. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00276 · Cited by 2,069 · see citing works
The NarrativeQA Reading Comprehension Challenge
Kočiský, T., Schwarz, J., Blunsom, P., Dyer, C., Hermann, K. M., Melis, G., & Grefenstette, E. (2018) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00023 · Cited by 596 · see citing works
S2ORC: The Semantic Scholar Open Research Corpus
Lo, K., Wang, L. L., Neumann, M., Kinney, R., & Weld, D. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.447 · Cited by 457 · see citing works
Datasets: A Community Library for Natural Language Processing
Lhoest, Q., Villanova del Moral, A., Jernite, Y., Thakur, A., von Platen, P., Patil, S., Chaumond, J., Drame, M., Plu, J., Tunstall, L., et al. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2021.emnlp-demo.21 · Cited by 343 · see citing works
QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarization
Zhong, M., Yin, D., Yu, T., Zaidi, A., Mutuma, M., Jha, R., Awadallah, A. H., Celikyilmaz, A., Liu, Y., Qiu, X., et al. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.472 · Cited by 178 · see citing works
Efficient Attentions for Long Document Summarization
Huang, L., Cao, S., Parulian, N., Ji, H., & Wang, L. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.112 · Cited by 146 · see citing works
A Practical Survey on Faster and Lighter Transformers
Fournier, Q., Caron, G. M., & Aloise, D. (2023) · ACM Computing Surveys · doi:10.1145/3586074 · Cited by 145 · see citing works
Do Long-Range Language Models Actually Use Long-Range Context?
Sun, S., Krishna, K., Mattarella-Micke, A., & Iyyer, M. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.62 · Cited by 43 · see citing works
Flagged first, then by citations
Research works printed but not linked: 23
- Loïc Barrault, Magdalena Biesialska, Ondřej Bojar, Marta R. Costa-jussà, Christian Federmann, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Matthias Huck, Eric Joanis, Tom Kocmi, Philipp Koehn, Chi-kiu Lo, Nikola Ljubešić, Christof Monz, Makoto Morishita, Masaaki Nagata, Toshi- aki Nakazawa, Santanu Pal, Matt Post, and Marcos Zampieri. 2020. Findings of the 2020 conference on machine translation (WMT20). In Proceedings of the Fifth Conference on Machine Translation, pages 1-55, Online. Association for Computational Lin- guistics.
- Loïc Barrault, Ondřej Bojar, Marta R. Costa-jussà, Christian Federmann, Mark Fishel, Yvette Gra- ham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, Christof Monz, Mathias Müller, Santanu Pal, Matt Post, and Marcos Zampieri. 2019. Findings of the 2019 conference on machine transla- tion (WMT19). In Proceedings of the Fourth Con- ference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 1-61, Florence, Italy. As- sociation for Computational Linguistics. · DOI
- Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877-1901.
- Jean Carletta, Simone Ashby, Sebastien Bourban, Mike Flynn, Mael Guillemot, Thomas Hain, Jaroslav Kadlec, Vasilis Karaiskos, Wessel Kraaij, Melissa Kronenthal, Guillaume Lathoud, Mike Lincoln, Agnes Lisowska, Iain McCowan, Wilfried Post, Dennis Reidsma, and Pierre Wellner. 2005. The ami meeting corpus: A pre-announcement. In In- ternational workshop on machine learning for multi- modal interaction, pages 28-39, Berlin, Heidelberg. Springer.
- Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Na- zli Goharian. 2018. A discourse-aware attention model for abstractive summarization of long docu- ments. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 2 (Short Papers), pages 615-621, New Orleans, Louisiana. Association for Computa- tional Linguistics. · DOI · Paper page
- Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A. Smith, and Matt Gardner. 2021. A dataset of information-seeking questions and answers an- chored in research papers. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, pages 4599-4610, On- line. Association for Computational Linguistics. · DOI · Paper page
- Charles J. Fillmore, Nancy Ide, Dan Jurafsky, and Catherine Macleod. 1998. An american national cor- pus: a proposal. In LREC.
- Adam Fisch, Alon Talmor, Robin Jia, Minjoon Seo, Eu- nsol Choi, and Danqi Chen. 2019. MRQA 2019 shared task: Evaluating generalization in reading 12015 comprehension. In Proceedings of the 2nd Work- shop on Machine Reading for Question Answering, pages 1-13, Hong Kong, China. Association for Computational Linguistics. · DOI
- Karl Moritz Hermann, Tomas Kocisky, Edward Grefen- stette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Informa- tion Processing Systems, volume 28.
- Nancy Ide and Keith Suderman. 2004. The Ameri- can national corpus first release. In Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC'04), Lisbon, Por- tugal. European Language Resources Association (ELRA).
- A. Janin, D. Baron, J. Edwards, D. Ellis, D. Gelbart, N. Morgan, B. Peskin, T. Pfau, E. Shriberg, A. Stol- cke, and C. Wooters. 2003. The icsi meeting corpus. In 2003 IEEE International Conference on Acous- tics, Speech, and Signal Processing, 2003. Proceed- ings. (ICASSP '03)., volume 1, pages I-I. · DOI
- Urvashi Khandelwal, He He, Peng Qi, and Dan Juraf- sky. 2018. Sharp nearby, fuzzy far away: How neu- ral language models use context. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 284-294, Melbourne, Australia. Association for Computational Linguistics. · DOI · Paper page
- Chin-Yew Lin. 2004. ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- tion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
- Richard Yuanzhe Pang, Alicia Parrish, Nitish Joshi, Nikita Nangia, Jason Phang, Angelica Chen, Vishakh Padmakumar, Johnny Ma, Jana Thompson, He He, and Samuel R. Bowman. 2021. QuAL- ITY: Question answering with long input texts, yes! arXiv preprint arXiv:2112.08608. · arXiv
- Ofir Press, Noah A. Smith, and Mike Lewis. 2021a. Shortformer: Better language modeling using shorter inputs. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 5493-5505, Online. Association for Computational Linguistics. · DOI · Paper page
- Ofir Press, Noah A. Smith, and Mike Lewis. 2021b. Train short, test long: Attention with linear biases enables input length extrapolation.
- Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't know: Unanswerable ques- tions for SQuAD. In Proceedings of the 56th An- nual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784- 789, Melbourne, Australia. Association for Compu- tational Linguistics. · DOI · Paper page
- Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natu- ral Language Processing, pages 2383-2392, Austin, Texas. Association for Computational Linguistics. · DOI · Paper page
- Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier. 2021. Efficient content-based sparse attention with routing transformers. Transac- tions of the Association for Computational Linguis- tics, 9:53-68. · DOI · Paper page
- Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. 2021. Long range arena : A benchmark for efficient trans- formers. In International Conference on Learning Representations.
- Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. 2020b. Efficient transformers: A survey. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems, volume 30.
- Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. Superglue: A stickier benchmark for general-purpose language un- derstanding systems. In Advances in Neural Infor- mation Processing Systems, volume 32.
- Alex Wang, Amanpreet Singh, Julian Michael, Fe- lix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis plat- form for natural language understanding. In Pro- ceedings of the 2018 EMNLP Workshop Black- boxNLP: Analyzing and Interpreting Neural Net- works for NLP, pages 353-355, Brussels, Belgium. Association for Computational Linguistics. · DOI · Paper page
Cited by
Sorted by citations
Levy, M., Jacoby, A., & Goldberg, Y. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.818 · Cited by 119 · see citing works
A Length-Extrapolatable Transformer
Sun, Y., Dong, L., Patra, B., Ma, S., Huang, S., Benhaim, A., Chaudhary, V., Song, X., & Wei, F. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.816 · Cited by 59 · see citing works
TRUE: Re-evaluating Factual Consistency Evaluation
Honovich, O., Aharoni, R., Herzig, J., Taitelbaum, H., Kukliansy, D., Cohen, V., Scialom, T., Szpektor, I., Hassidim, A., & Matias, Y. (2022) · Proceedings of the Second DialDoc Workshop on Document-grounded Dialogue and Conversational Question Answering · doi:10.18653/v1/2022.dialdoc-1.19 · Cited by 56 · see citing works
Hutchins, D., Schlag, I., Wu, Y., Dyer, E., & Neyshabur, B. (2022) · arXiv · doi:10.48550/arxiv.2203.07852 · Cited by 43 · see citing works
Efficient long-text understanding with short-text models
Ivgi, M., Shaham, U., & Berant, J. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00547 · Cited by 42 · see citing works
Self-citationZeroSCROLLS: A zero-shot benchmark for long text understanding
Shaham, U., Ivgi, M., Efrat, A., Berant, J., & Levy, O. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.536 · Cited by 23 · see citing works
Self-citationL-Eval: Instituting Standardized Evaluation for Long Context Language Models
An, C., Gong, S., Zhong, M., Zhao, X., Li, M., Zhang, J., Kong, L., & Qiu, X. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.776 · Cited by 18 · see citing works
L-Eval: Instituting Standardized Evaluation for Long Context Language Models
An, C., Gong, S., Zhong, M., Zhao, X., Li, M., Zhang, J., Kong, L., & Qiu, X. (2023) · arXiv · doi:10.48550/arxiv.2307.11088 · Cited by 6 · see citing works
“Meanwhile, almost all previous long-sequence text generation benchmarks relied primarily on n-gram matching metrics (Zhang et al., 2023; Shaham et al., 2022), such as ROUGE (Lin, 2004).”
BackgroundMethodInfluentialBABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack
Kuratov, Y., Bulatov, A., Anokhin, P., Rodkin, I., Sorokin, D., Sorokin, A., & Burtsev, M. (2024) · arXiv · doi:10.48550/arxiv.2406.10149 · Cited by 19 · see citing works
“Scrolls and ZeroSCROLLS (Shaham et al., 2022, 2023) consist of QA, classification, summarization tasks and have higher average lengths ranging from 1.7k to 49.3k tokens.”
BackgroundAutoMix: Automatically Mixing Language Models
Aggarwal, P., Madaan, A., Anand, A., Potharaju, S. P., Mishra, S., Zhou, P., Gupta, A., Rajagopal, D., Kappaganthu, K., Yang, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.12963 · Cited by 19 · see citing works
Self-citationDong, Z., Tang, T., Li, J., Zhao, W. X., & Wen, J.-R. (2023) · arXiv · doi:10.48550/arxiv.2309.13345 · Cited by 7 · see citing works
“Recently, with LLM drawing a great deal of attention, Ze-roSCROLLS (Shaham et al., 2023), L-Eval (An et al., 2023), and LongBench (Bai et al., 2023) were released, focusing on the zero-shot long text comprehension ability of LLMs.”
MethodA Slice and Dice Approach to Accelerate Compound Sparse Attention on GPU
Li, H., Choi, J., & Ahn, J. H. (2022) · 2022 IEEE International Symposium on Workload Characterization (IISWC) · doi:10.1109/iiswc55918.2022.00019 · Cited by 3 · see citing works
HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughly
Yen, H., Gao, T., Hou, M., Ding, K., Fleischer, D., Izsak, P., Wasserblat, M., & Chen, D. (2024) · arXiv · Cited by 2 · see citing works
“As discussed in the main paper, existing benchmarks either are limit to relatively short context lengths (Shaham et al., 2022; 2023; Li et al., 2024a; An et al., 2024; Bai et al., 2024; Dong et al., 2024) or lack rigorous evaluation methods (Zhang et al., 2024; Yuan et al., 2024).”
BackgroundMemory-Augmented Architecture for Long-Term Context Handling in Large Language Models
Shinwari, H. U. K., & Usama, M. (2026) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2026.3701680
“…open-domain conversation; multisession chat (MSC) [16] and BlendedSkillTalk [17], which test for consistency across conversational sessions and the ability to blend multiple skills; and LongBench [18] and SCROLLS [19], which assess performance on long-document understanding and question answering.”
VehicleMemBench: An Executable Benchmark for Multi-User Long-Term Memory in In-Vehicle Agents
Chen, Y., Xu, Y., Ding, X., Fang, X., Liu, S., Lin, L., Zhang, Q., Li, Y., Liu, Q., & Xu, T. (2026) · arXiv · doi:10.48550/arxiv.2603.23840
“Early work characterized memory capabilities through evaluations of long-context and long-range dependencies [Shaham et al., 2022], focusing on sequence understanding and information retention.”
Zhang, Y., Liu, F., Shan, Y., Huang, X., Yang, X., Zhu, Y., Cheng, X., Liu, C., Zeng, K., Zhang, T. J., et al. (2026) · arXiv · doi:10.48550/arxiv.2603.01045
“Existing benchmarks like SCROLLS (Shaham et al., 2022), LongBench (Bai et al., 2024), and ∞ Bench (Zhang et al., 2024b) effectively evaluate single-agent retrieval but over-look the paradigm of distributed collaboration .”
- Authors
- Shaham, U., Segal, E., Ivgi, M., Efrat, A., Yoran, O., Haviv, A., Gupta, A., Xiong, W., Geva, M., Berant, J., & Levy, O.
- Affiliations
- Tel Aviv University (IL)
- IBM (United States) (US)
- Venue
- Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing
- Published
- 2022
- Pages
- 12007-12021
- OpenAlex
- W4385573804
- Cited by
- 40
- References
- 11
- Citations by year
- 2026: 4 · 2025: 6 · 2024: 4 · 2023: 17 · 2022: 9
- FWCI
- 4.98
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- Funders
- European Commission
- Tel Aviv University
- H2020 European Research Council (802800)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Keywords
- natural language understanding
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
