Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Models
Open accessAbstract
This paper explores the impact of extending input lengths on the capabilities of Large Language Models (LLMs). Despite LLMs advancements in recent times, their performance consistency across different input lengths is not well understood. We investigate this aspect by introducing a novel QA reasoning framework, specifically designed to assess the impact of input length. We isolate the effect of input length using multiple versions of the same sample, each being extended with padding of different lengths, types and locations. Our findings show a notable degradation in LLMs' reasoning performance at much shorter input lengths than their technical maximum. We show that the degradation trend appears in every version of our dataset, although at different intensities. Additionally, our study reveals that the traditional metric of next word prediction correlates negatively with performance of LLMs' on our reasoning dataset. We analyse our results and identify failure modes that can serve as useful guides for future research, potentially informing strategies to address the limitations observed in LLMs.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Simply increasing input token lengths results in greater misalignment.
“given that simply increasing input token lengths results in greater misalignment [9]”
From Evaluating Metrics for Safety with LLM-as-Judges (2025) · cites this paper as [9]
Section 3.3 describes increasing input lengths while keeping the underlying task fixed. Section 7 reports that longer inputs led some models to disregard answer-format and reasoning-order instructions. That supports a narrower claim about instruction-following failures, but the paper does not establish greater misalignment in general. (LLM, conf=0.84)
Experimental — accuracy not yet measuredAdditional input length can impair reasoning in some settings.
“Additional input length can also impair reasoning in some settings (Levy et al., 2024).”
From Beyond the Context Window: A Cost-Performance Analysis of Fact-Based Memory vs. Long-Context LLMs for Persistent Agents (2026) · cites this paper as Levy et al., 2024
Recent works such as FlenQA have demonstrated degrading performance when increasing task input length to just a few thousand tokens.
“In fact, recent works, such as FlenQA (Levy et al., 2024), have demonstrated degrading performance when increasing their task input length to just a few thousand tokens.”
From RULER: What's the Real Context Size of Your Long-Context Language Models? (2024) · cites this paper as Levy et al., 2024
Prior works do not study how results on synthetic tasks transfer to real applications.
“However, they do not study how results on synthetic tasks transfer to real applications.”
From HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughly (2024) · cites this paper as Levy et al., 2024
The Related Work section (page unavailable) discusses long-input benchmarks and an inconsistent correlation between next-word prediction and downstream performance, but does not say whether prior studies examined transfer from synthetic tasks to real applications. The Limitations section (page unavailable) discusses the scope of this paper’s own tasks, not that claim about prior work. Searches for synthetic-task transfer and real-world applications found no passage addressing it. (LLM, conf=0.72)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
No references could be resolved yet.
Research works printed but not linked: 35
- Chen An, Shansan Gong, Ming Zhong, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. 2023a. L- eval: Instituting standardized evaluation for long con- text language models. ArXiv, abs/2307.11088. · arXiv
- Rohan Anil, Sebastian Borgeaud, Yonghui Wu, and Gemini Team Google. 2023. Gemini: A fam- ily of highly capable multimodal models. ArXiv, abs/2312.11805. · arXiv
- Yushi Bai, Xin Lv, Jiajie Zhang, Hong Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Ao- han Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2023. Longbench: A bilingual, multitask benchmark for long context understanding. ArXiv, abs/2308.14508. · arXiv
- Jifan Chen and Greg Durrett. 2019. Understanding dataset design choices for multi-hop reasoning. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers).
- Peter Clark, Oyvind Tafjord, and Kyle Richardson. 2021. Transformers as soft reasoners over language. In Pro- ceedings of the Twenty-Ninth International Confer- ence on International Joint Conferences on Artificial Intelligence, pages 3882-3890.
- Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. · arXiv
- Ishita Dasgupta, Andrew K Lampinen, Stephanie CY Chan, Antonia Creswell, Dharshan Kumaran, James L McClelland, and Felix Hill. 2022. Lan- guage models show human-like content effects on reasoning. arXiv preprint arXiv:2207.07051. · arXiv
- Alexios Gidiotis and Grigorios Tsoumakas. 2020. A divide-and-conquer approach to the summarization of long documents. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28:3029-3040.
- Ari Holtzman, Peter West, and Luke Zettlemoyer. 2023. Generative models as a complex systems science: How can we make sense of large language model behavior? arXiv preprint arXiv:2308.00189. · arXiv
- Jie Huang and Kevin Chen-Chuan Chang. 2022. To- wards reasoning in large language models: A survey. arXiv preprint arXiv:2212.10403. · arXiv
- Alon Jacovi, Avi Caciularu, Omer Goldman, and Yoav Goldberg. 2023. Stop uploading test data in plain text: Practical strategies for mitigating data contami- nation by evaluation benchmarks. In Proceedings of the 2023 Conference on Empirical Methods in Natu- ral Language Processing, pages 5075-5084, Singa- pore. Association for Computational Linguistics. · DOI · Paper page
- Mingyu Jin, Qinkai Yu, Haiyan Zhao, Wenyue Hua, Yanda Meng, Yongfeng Zhang, Mengnan Du, et al. 2024. The impact of reasoning step length on large language models. arXiv preprint arXiv:2401.04925. · arXiv
- Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large lan- guage models are zero-shot reasoners. Advances in neural information processing systems, 35:22199- 22213.
- Mosh Levy, Shauli Ravfogel, and Yoav Goldberg. 2023. Guiding llm to fool itself: Automatically manipulat- ing machine reading comprehension shortcut triggers. In Findings of the Association for Computational Lin- guistics: EMNLP 2023, pages 8495-8505.
- Jiaqi Li, Mengmeng Wang, Zilong Zheng, and Muhan Zhang. 2023. Loogle: Can long-context lan- guage models understand long contexts? ArXiv, abs/2311.04939. · arXiv
- Hong Liu, Sang Michael Xie, Zhiyuan Li, and Tengyu Ma. 2023a. Same pre-training loss, better down- stream: Implicit bias matters for language models. In International Conference on Machine Learning, pages 22188-22214.
- PMLR.
- Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paran- jape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2023b. Lost in the middle: How lan- guage models use long contexts. arXiv preprint arXiv:2307.03172. · arXiv
- Yang Liu, Chenguang Zhu, and Michael Zeng. 2022. End-to-end segmentation-based news summarization. In Findings of the Association for Computational Linguistics: ACL 2022, pages 544-554.
- Yuning Mao, Pengcheng He, Xiaodong Liu, Yelong Shen, Jianfeng Gao, Jiawei Han, and Weizhu Chen. 2020. Generation-augmented retrieval for open- domain question answering. In Annual Meeting of the Association for Computational Linguistics.
- Sewon Min, Eric Wallace, Sameer Singh, Matt Gardner, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2019. Compositional questions do not necessitate multi-hop reasoning. In Annual Meeting of the Association for Computational Linguistics. OpenAI. 2023. Gpt-4 technical report.
- Oscar Sainz, Jon Campos, Iker García-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle, and Eneko Agirre. 2023. Nlp evaluation in trouble: On the need to measure llm data contamination for each benchmark. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10776-10787.
- Uri Shaham, Maor Ivgi, Avia Efrat, Jonathan Be- rant, and Omer Levy. 2023. Zeroscrolls: A zero- shot benchmark for long text understanding. arXiv preprint arXiv:2305.14196. · arXiv
- Uri Shaham, Elad Segal, Maor Ivgi, Avia Efrat, Ori Yoran, Adi Haviv, Ankit Gupta, Wenhan Xiong, Mor Geva, Jonathan Berant, et al. 2022. Scrolls: Stan- dardized comparison over long language sequences. In Proceedings of the 2022 Conference on Empiri- cal Methods in Natural Language Processing, pages 12007-12021.
- Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H Chi, Nathanael Schärli, and Denny Zhou. 2023. Large language models can be easily distracted by irrelevant context. In Inter- national Conference on Machine Learning, pages 31210-31227.
- PMLR.
- Koustuv Sinha, Shagun Sodhani, William L. Hamilton, and Joelle Pineau. 2018. Compositional language understanding with text-based relational reasoning. ArXiv, abs/1811.02959. · arXiv
- Yi Tay, Mostafa Dehghani, Samira Abnar, Hyung Won Chung, William Fedus, Jinfeng Rao, Sharan Narang, Vinh Q Tran, Dani Yogatama, and Donald Metzler. 15348 2022. Scaling laws vs model architectures: How does inductive bias influence scaling? arXiv preprint arXiv:2207.10551. · arXiv
- Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits rea- soning in large language models. Advances in Neural Information Processing Systems, 35:24824-24837.
- Jason Weston, Antoine Bordes, Sumit Chopra, Alexan- der M Rush, Bart Van Merriënboer, Armand Joulin, and Tomas Mikolov. 2016. Towards ai-complete question answering: A set of prerequisite toy tasks. In 4th International Conference on Learning Repre- sentations, ICLR 2016.
- Ruben Wolhandler, Arie Cattan, Ori Ernst, and Ido Da- gan. 2022. How "multi" is multi-document summa- rization? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5761-5769.
- Mengzhou Xia, Mikel Artetxe, Chunting Zhou, Xi Vic- toria Lin, Ramakanth Pasunuru, Danqi Chen, Luke Zettlemoyer, and Ves Stoyanov. 2022. Training tra- jectories of language models across scales. arXiv preprint arXiv:2212.09803. · arXiv
- Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. arXiv preprint arXiv:2305.10601. · arXiv
- Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2022. Large language models are human-level prompt engineers. arXiv preprint arXiv:2211.01910. · arXiv
- Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhut- dinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In The IEEE International Con- ference on Computer Vision (ICCV).
Cited by
Sorted by citations
RULER: What's the Real Context Size of Your Long-Context Language Models?
Hsieh, C.-P., Sun, S., Kriman, S., Acharya, S., Rekesh, D., Jia, F., Zhang, Y., & Ginsburg, B. (2024) · arXiv · Cited by 13 · see citing works
“…including fact reasoning (Kuratov et al., 2024; Karpinska et al., 2024), long-range discourse modeling (Sun et al., 2022), question answering (Levy et al., 2024; Yuan et al., 2024), many-shot in-context learning (Agarwal et al., 2024; Bertsch et al., 2024; Xu et al., 2024b), and code…”
BackgroundNoLiMa: Long-Context Evaluation Beyond Literal Matching
Modarressi, A., Deilamsalehy, H., Dernoncourt, F., Bui, T., Rossi, R. A., Yoon, S., & Schütze, H. (2025) · arXiv · doi:10.48550/arxiv.2502.05167 · Cited by 3 · see citing works
“These enhancements include introducing multiple needles, incorporating additional distractor material, and interconnecting facts to necessitate in-context reasoning (e.g., fact-chaining) (Hsieh et al., 2024; Levy et al., 2024; Kuratov et al., 2024).”
Toward conversational agents with context and time sensitive long-term memory
Alonso, N., Figliolia, T., Ndirango, A., & Millidge, B. (2024) · arXiv · doi:10.48550/arxiv.2406.00057 · Cited by 2 · see citing works
HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughly
Yen, H., Gao, T., Hou, M., Ding, K., Fleischer, D., Izsak, P., Wasserblat, M., & Chen, D. (2024) · arXiv · Cited by 2 · see citing works
“Recent works have expanded upon it and designed new procedures to test different aspects of LCLMs (Hsieh et al., 2024; Li et al., 2024b; Levy et al., 2024; Arora et al., 2023; Laban et al., 2024; Goldman et al., 2024).”
BackgroundA Multi-Agent Framework for Accessible Web Code Generation
Aljedaani, W., Shaik, M. A., & Aljohani, A. (2026) · Proceedings of the 23rd International Web for All Conference · doi:10.1145/3800424.3800453 · Cited by 1 · see citing works
Evaluating Metrics for Safety with LLM-as-Judges
Clegg, K., Hawkins, R., Habli, I., & Lawton, T. (2025) · arXiv · doi:10.48550/arxiv.2512.15617
Pollertlam, N., & Kornsuwannawit, W. (2026) · arXiv · doi:10.48550/arxiv.2603.04814
- Authors
- Levy, M., Jacoby, A., & Goldberg, Y.
- Affiliations
- Bar-Ilan University (IL)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- Pages
- 15339-15353
- ISSN
- 0736-587X (ISSN-L)
- arXiv
- 2402.14848
- OpenAlex
- W4402683949
- Cited by
- 119
- References
- 0
- Citations by year
- 2026: 48 · 2025: 57 · 2024: 14
- FWCI
- 35.12
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- European Commission (HORIZON2020)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- large language models, reasoning performance, question answering, padding, next-word prediction, performance degradation, failure modes
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
