Do Long-Range Language Models Actually Use Long-Range Context?
Open accessAbstract
Language models are generally trained on short, truncated input sequences, which limits their ability to use discourse-level information present in long-range context to improve their predictions.Recent efforts to improve the efficiency of self-attention have led to a proliferation of long-range Transformer language models, which can process much longer sequences than models of the past.However, the ways in which such models take advantage of the longrange context remain unclear.In this paper, we perform a fine-grained analysis of two longrange Transformer language models (including the Routing Transformer, which achieves state-of-the-art perplexity on the PG-19 longsequence LM benchmark dataset) that accept input sequences of up to 8K tokens.Our results reveal that providing long-range context (i.e., beyond the previous 2K tokens) to these models only improves their predictions on a small set of tokens (e.g., those that can be copied from the distant context) and does not help at all for sentence-level prediction tasks.Finally, we discover that PG-19 contains a variety of different document types and domains, and that long-range context helps most for literary novels (as opposed to textbooks or magazines).
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Transformers process natural language as a sequence on finite context windows, and global relationships among words beyond these windows cannot be completely modeled via sequence processing only.
“Transformers process natural language as a sequence on finite context windows; however, global relationships among words beyond these windows cannot be completely modeled via sequence processing only.”
From Emotion Classification With Visibility Graphs (2025) · cites this paper as [12]
The Introduction says Transformer language models have maximum sequence lengths and that small context windows prevent models from using long-range discourse structure. The Conclusion reports that the two models studied rarely take advantage of long-range context. The paper does not establish the broader claim that sequence processing alone cannot completely model relationships beyond a window. (LLM, conf=0.85)
Experimental — accuracy not yet measuredPretrained-only base LLMs are known to show biased attention for more recent tokens (the end of inputs), especially when evaluating next-word prediction of contiguous text.
“Pretrained-only base LLMs are known to show biased attention for more recent tokens (that is, the end of inputs), especially when evaluating next-word prediction of contiguous text52.”
From A psychometric framework for evaluating and shaping personality traits in large language models (2025) · cites this paper as [52]
Section 3 reports that overall next-token perplexity plateaus beyond a 2K-token prefix, and Section 4 finds that the Routing Transformer relies on the most recent occurrences of repeated tokens for prediction. Section 5 confirms that the token-level evaluations use ground-truth contiguous prefixes. These results support a preference for recent context in the models studied, but do not establish biased attention weights across pretrained-only base LLMs generally. (LLM, conf=0.84)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works
Effective Approaches to Attention-based Neural Machine Translation
Luong, T., Pham, H., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1166 · Cited by 8,382 · see citing works
Transformer-XL: Attentive language models beyond a fixed-length context
Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q., & Salakhutdinov, R. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1285 · Cited by 3,169 · see citing works
Kronenfeld, D. B. (1978) · Language · doi:10.2307/412850 · Cited by 3,094 · see citing works
Scripts, Plans, Goals, and Understanding: An Inquiry into Human Knowledge Structures
Hunt, E. B., Schank, R. C., & Abelson, R. P. (1979) · The American Journal of Psychology · doi:10.2307/1421499 · Cited by 2,967 · see citing works
Narrative Analysis: Oral Versions of Personal Experience1
Labov, W., & Waletzky, J. (1997) · Journal of Narrative and Life History · doi:10.1075/jnlh.7.02nar · Cited by 2,919 · see citing works
Longformer: The Long-Document Transformer
Beltagy, I., Peters, M. E., & Cohan, A. (2020) · arXiv · doi:10.48550/arxiv.2004.05150 · Cited by 2,281 · see citing works
Centering: A Framework for Modelling the Local Coherence of Discourse,
Grosz, B. J., Joshi, A. K., & Weinstein, S. (1995) · doi:10.21236/ada324949 · Cited by 1,793 · see citing works
Reformer: The Efficient Transformer
Kitaev, N., Kaiser, Ł., & Levskaya, A. (2020) · arXiv · doi:10.48550/arxiv.2001.04451 · Cited by 1,435 · see citing works
Big Bird: Transformers for Longer Sequences
Zaheer, M., Guruganesh, G., Dubey, A., Ainslie, J., Alberti, C., Ontanon, S., Pham, P., Ravula, A., Wang, Q., Yang, L., et al. (2020) · arXiv · doi:10.48550/arxiv.2007.14062 · Cited by 1,349 · see citing works
The Curious Case of Neural Text Degeneration
Holtzman, A., Buys, J., Du, L., Forbes, M., & Choi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09751 · Cited by 1,304 · see citing works
Generating Long Sequences with Sparse Transformers
Child, R., Gray, S., Radford, A., & Sutskever, I. (2019) · arXiv · doi:10.48550/arxiv.1904.10509 · Cited by 1,063 · see citing works
Hobbs, J. R. (1979) · Cognitive Science · doi:10.1207/s15516709cog0301_4 · Cited by 907 · see citing works
Linformer: Self-Attention with Linear Complexity
Wang, S., Li, B. Z., Khabsa, M., Fang, H., & Ma, H. (2020) · arXiv · doi:10.48550/arxiv.2006.04768 · Cited by 867 · see citing works
Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention
Katharopoulos, A., Vyas, A., Pappas, N., & Fleuret, F. (2020) · arXiv · doi:10.48550/arxiv.2006.16236 · Cited by 827 · see citing works
SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference
Zellers, R., Bisk, Y., Schwartz, R., & Choi, Y. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1009 · Cited by 712 · see citing works
Rethinking Attention with Performers
Choromanski, K., Likhosherstov, V., Dohan, D., Song, X., Gane, A., Sarlos, T., Hawkins, P., Davis, J., Mohiuddin, A., Kaiser, L., et al. (2020) · arXiv · doi:10.48550/arxiv.2009.14794 · Cited by 695 · cited 1× · see citing works
“Our work examines recent advances in efficient Transformer variants (Sukhbaatar et al., 2019; Ki-taev et al., 2020; Choromanski et al., 2021; Tay et al., 2021; Katharopoulos et al., 2020; Wang et al., 2020; Wu et al., 2020) that accept longer sequences than prior approaches.”
BackgroundA maximum entropy approach to adaptive statistical language modelling
Rosenfeld, R. (1996) · Computer Speech & Language · doi:10.1006/csla.1996.0011 · Cited by 546 · see citing works
Efficient Content-Based Sparse Attention with Routing Transformers
Roy, A., Saffar, M., Vaswani, A., & Grangier, D. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00353 · Cited by 509 · see citing works
(2020) · arXiv (Cornell University) · Cited by 461 · see citing works
Sharp nearby, fuzzy far away: How neural language models use context
Khandelwal, U., He, H., Qi, P., & Jurafsky, D. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1027 · Cited by 303 · see citing works
Adaptive Attention Span in Transformers
Sukhbaatar, S., Grave, É., Bojanowski, P., & Joulin, A. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1032 · Cited by 286 · see citing works
Lite Transformer with Long-Short Range AttentionPreprint
Wu, Z., Liu, Z., Lin, J., Lin, Y., & Han, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.11886 · Cited by 279 · see citing works
Improving the Transformer Translation Model with Document-Level Context
Zhang, J.-C., Luan, H.-B., Sun, M., Zhai, F.-F., Xu, J.-F., Zhang, M., & Liu, Y. (2018) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1049 · Cited by 274 · see citing works
Neural Text Generation with Unlikelihood Training
Welleck, S., Kulikov, I., Roller, S., Dinan, E., Cho, K., & Weston, J. (2019) · arXiv · doi:10.48550/arxiv.1908.04319 · Cited by 254 · see citing works
Synthesizer: Rethinking Self-Attention in Transformer Models
Tay, Y., Bahri, D., Metzler, D., Juan, D.-C., Zhao, Z., & Zheng, C. (2020) · arXiv · doi:10.48550/arxiv.2005.00743 · Cited by 251 · see citing works
Adaptively Sparse Transformers
Correia, G. M., Niculae, V., & Martins, A. F. T. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1223 · Cited by 211 · see citing works
Compressive Transformers for Long-Range Sequence Modelling
Rae, J. W., Potapenko, A., Jayakumar, S. M., & Lillicrap, T. P. (2019) · arXiv · doi:10.48550/arxiv.1911.05507 · Cited by 205 · see citing works
On Exposure Bias, Hallucination and Domain Shift in Neural Machine Translation
Wang, C., & Sennrich, R. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.326 · Cited by 118 · see citing works
Hurdles to Progress in Long-form Question Answering
Krishna, K., Roy, A., & Iyyer, M. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.393 · Cited by 114 · see citing works
Self-citationNeural Text Generation With Unlikelihood Training
Welleck, S., Kulikov, I., Roller, S., Dinan, E., Cho, K., & Weston, J. (2020) · International Conference on Learning Representations · Cited by 106 · see citing works
Learning schemata for natural language processing
Mooney, R. J., & DEJONG, G. F. (1985) · Cited by 100 · see citing works
Larger-Context Language Modelling with Recurrent Neural Network
Wang, T., & Cho, K. (2016) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p16-1125 · Cited by 95 · see citing works
Local Self-Attention over Long Text for Efficient Document Retrieval
Hofstätter, S., Zamani, H., Mitra, B., Craswell, N., & Hanbury, A. (2020) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3397271.3401224 · Cited by 92 · see citing works
Document Context Language ModelsPreprint
Ji, Y., Cohn, T., Kong, L., Dyer, C., & Eisenstein, J. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.03962 · Cited by 57 · see citing works
Shortformer: Better language modeling using shorter inputs
Press, O., Smith, N. A., & Lewis, M. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.427 · Cited by 55 · see citing works
Zhang, P., Chen, B., Ge, N., & Fan, K. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.81 · Cited by 29 · see citing works
Do Transformers Need Deep Long-Range Memory?
Rae, J. W., & Razavi, A. S. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.672 · Cited by 26 · see citing works
Prediction with a short memory
Sharan, V., Kakade, S. M., Liang, P., & Valiant, G. J. (2018) · ACM Symposium on Theory of Computing (STOC) · doi:10.1145/3188745.3188954 · Cited by 24 · see citing works
Synthesizer: Rethinking Self-Attention for Transformer Models
Tay, Y., Bahri, D., Metzler, D., Juan, D.-C., Zhao, Z., & Zheng, C. (2021) · International Conference on Machine Learning · Cited by 18 · see citing works
Context Analysis for Pre-trained Masked Language Models
Lai, Y.-A., Lalwani, G., & Zhang, Y. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.338 · Cited by 11 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805
Flagged first, then by citations
3 of the 46 references couldn’t be matched to a known paper.
Research works printed but not linked: 2
- Jane Grimshaw. 1990. Argument structure. the MIT Press.
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
Cited by
Sorted by citations
Lost in the Middle: How Language Models Use Long Contexts
Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00638 · Cited by 1,612 · see citing works
“Tay et al. (2021) propose the Long Range Arena, which evaluates long-context models on a variety of natural language, visual reasoning, and synthetic tasks.”
BackgroundThe power of noise: Redefining retrieval for rag systems
Cuconasu, F., Trappolini, G., Siciliano, F., Filice, S., Campagnano, C., Maarek, Y., Tonellotto, N., & Silvestri, F. (2024) · Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3626772.3657834 · Cited by 236 · see citing works
Personality Traits in Large Language ModelsPreprint
Serapio‐García, G., Safdari, M., Crepy, C., Sun, L., Fitz, S., Abdulhai, M., Faust, A., & Matarić, M. J. (2023) · Research Square · doi:10.21203/rs.3.rs-3296728/v1 · Cited by 137 · see citing works
GenSLMs: Genome-scale language models reveal SARS-CoV-2 evolutionary dynamics
Zvyagin, M., Brace, A., Hippe, K., Deng, Y., Zhang, B., Bohorquez, C. O., Clyde, A., Kale, B., Perez-Rivera, D., Ma, H., et al. (2023) · The International Journal of High Performance Computing Applications · doi:10.1177/10943420231201154 · Cited by 120 · see citing works
GenSLMs: Genome-scale language models reveal SARS-CoV-2 evolutionary dynamics
Zvyagin, M., Brace, A., Hippe, K., Deng, Y., Zhang, B., Bohorquez, C. O., Clyde, A., Kale, B., Perez-Rivera, D., Ma, H., et al. (2022) · doi:10.1101/2022.10.10.511571 · Cited by 120 · see citing works
Hutchins, D., Schlag, I., Wu, Y., Dyer, E., & Neyshabur, B. (2022) · arXiv · doi:10.48550/arxiv.2203.07852 · Cited by 43 · see citing works
SCROLLS: Standardized CompaRison Over Long Language Sequences
Shaham, U., Segal, E., Ivgi, M., Efrat, A., Yoran, O., Haviv, A., Gupta, A., Xiong, W., Geva, M., Berant, J., et al. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.823 · Cited by 40 · see citing works
SemantiCodec: An Ultra Low Bitrate Semantic Audio Codec for General Sound
Liu, H., Xu, X., Yuan, Y., Wu, M., Wang, W., & Plumbley, M. D. (2024) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2024.3506286 · Cited by 40 · see citing works
Statler: State-Maintaining Language Models for Embodied Reasoning
Yoneda, T., Fang, J., Li, P., Zhang, H., Jiang, T., Lin, S., Picker, B., Yunis, D., Mei, H., & Walter, M. R. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10610634 · Cited by 41 · see citing works
RankGen: Improving Text Generation with Large Ranking Models
Krishna, K., Chang, Y., Wieting, J., & Iyyer, M. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.15 · Cited by 30 · see citing works
Self-citationA psychometric framework for evaluating and shaping personality traits in large language models
Serapio-García, G., Safdari, M., Crepy, C., Sun, L., Fitz, S., Romero, P., Abdulhai, M., Faust, A., & Matarić, M. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-025-01115-6 · Cited by 30 · see citing works
Language models can identify enzymatic binding sites in protein sequences
Teukam, Y. G. N., Dassi, L. K., Manica, M., Probst, D., Schwaller, P., & Laino, T. (2024) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2024.04.012 · Cited by 31 · see citing works
Efficient Long-Range Transformers: You Need to Attend More, but Not Necessarily at Every Layer
Zhang, Q., Ram, D., Hawkins, C., Zha, S., & Zhao, T. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.183 · Cited by 23 · see citing works
OctoPack: Instruction Tuning Code Large Language Models
Muennighoff, N., Liu, Q., Zebaze, A., Zheng, Q., Hui, B., Zhuo, T. Y., Singh, S., Tang, X., von Werra, L., & Longpre, S. (2023) · arXiv · doi:10.48550/arxiv.2308.07124 · Cited by 17 · see citing works
“, 2023d) and the necessary long context windows (Dai et al., 2019; Press et al., 2021; Sun et al., 2021; Dao et al., 2022; Peng et al., 2023; Liu et al., 2023c; Chen et al., 2023b).”
BackgroundCoherence boosting: When your pretrained language model is not paying enough attention
Malkin, N., Wang, Z., & Jojić, N. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.565 · Cited by 20 · see citing works
The NLP Task Effectiveness of Long-Range Transformers
Qin, G., Feng, Y., & Van Durme, B. (2023) · Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.273 · Cited by 14 · see citing works
Predicting Attention Sparsity in Transformers
Treviso, M., Góis, A., Fernandes, P., Fonseca, E., & Martins, A. F. T. (2022) · Workshop on Structured Prediction for NLP (SPNLP) · doi:10.18653/v1/2022.spnlp-1.7 · Cited by 12 · see citing works
Retrieval-Pretrained Transformer: Long-range Language Modeling with Self-retrieval
Rubin, O., & Berant, J. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00693 · Cited by 19 · see citing works
The 6th International Workshop on Narrative Extraction from Texts: Text2Story 2023
Campos, R., Jorge, A., Jatowt, A., Bhatia, S., & Litvak, M. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-28241-6_40 · Cited by 9 · see citing works
Language Models Can Identify Enzymatic Active Sites in Protein SequencesPreprint
Teukam, Y. G. N., Dassi, L. K., Manica, M., Schwaller, P., Probst, D., & Laino, T. (2023) · SSRN Electronic Journal · doi:10.26434/chemrxiv-2021-m20gg-v3 · Cited by 8 · see citing works
ChapterBreak: A Challenge Dataset for Long-Range Language Models
Sun, S., Thai, K., & Iyyer, M. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.271 · Cited by 7 · see citing works
Self-citationL-Eval: Instituting Standardized Evaluation for Long Context Language Models
An, C., Gong, S., Zhong, M., Zhao, X., Li, M., Zhang, J., Kong, L., & Qiu, X. (2023) · arXiv · doi:10.48550/arxiv.2307.11088 · Cited by 6 · see citing works
“In language domain, previous work on LCLMs tends to report PPL to evaluate language models (Su et al., 2022; Peng et al., 2023b) on longer context.”
BackgroundThe 7th International Workshop on Narrative Extraction from Texts: Text2Story 2024
Campos, R., Jorge, A., Jatowt, A., Bhatia, S., & Litvak, M. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-56069-9_52 · Cited by 6 · see citing works
LaMemo: Language Modeling with Look-Ahead Memory
Ji, H., Zhang, R., Yang, Z., Hu, Z., & Huang, M. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.422 · Cited by 5 · see citing works
Oh, B.-D., & Schuler, W. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.562 · Cited by 5 · see citing works
Dong, Z., Tang, T., Li, J., Zhao, W. X., & Wen, J.-R. (2023) · arXiv · doi:10.48550/arxiv.2309.13345 · Cited by 7 · see citing works
“However, most tokens only needed modeling a limited context instead of long-term relations (Sun et al., 2021).”
BackgroundLong-Range Language Modeling with Selective Cache
Huang, X., & Hollenstein, N. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.321 · Cited by 3 · see citing works
Aligning LLMs to Fully Utilize the Cross-file Context in Repository-level Code Completion
Li, J., Zhu, H., Liu, H., Xianjie, S., Zong, H., Dong, Y., Zhang, K., Jiang, S., Jin, Z. G., & Li, G. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00125 · Cited by 3 · see citing works
Model Criticism for Long-Form Text Generation
Deng, Y., Kuleshov, V., & Rush, A. M. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.815 · Cited by 4 · see citing works
A unifying computational account of temporal context effects in language across the human cortexPreprint
Vo, V. A., Jain, S., Beckage, N., Chien, H.-Y. S., Obinwa, C., & Huth, A. G. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.08.03.551886 · Cited by 3 · see citing works
Emotion Classification With Visibility Graphs
Şimşek, E., Topcu, A., Koç, E., Saritas, E. U., & Koç, A. (2025) · IEEE Signal Processing Letters · doi:10.1109/lsp.2025.3567033 · Cited by 2 · see citing works
“More specifically, sequential models process text as token sequences within fixed context windows, limiting their ability to capture word-level dependencies that span beyond those boundaries [12], [13].”
Holtzman, A., West, P., & Zettlemoyer, L. (2025) · Journal of Social Computing · doi:10.23919/jsc.2025.0009 · Cited by 14 · see citing works
Discovering technology-aided possibilities for automatic analysis of science teacher questions
Lehesvuori, S., Urrutia, F., Heilala, V., Araya, R., & Hämäläinen, R. (2025) · Educational Technology Research and Development · doi:10.1007/s11423-025-10545-3 · Cited by 2 · see citing works
A method for semantic textual similarity on long texts
Zatarain, O. A., González-Castolo, J. C., & Ramos-Cabral, S. (2025) · PeerJ Computer Science · doi:10.7717/peerj-cs.3202 · Cited by 2 · see citing works
GEM-RAG: Graphical Eigen Memories for Retrieval Augmented Generation
Rappazzo, B., Wang, Y., Ferber, A., & Gomes, C. P. (2024) · International Conference on Machine Learning and Applications · doi:10.1109/icmla61862.2024.00196 · Cited by 2 · see citing works
Black-box language model explanation by context length probing
Cífka, O., & Antoine Liutkus, A. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-short.92 · Cited by 2 · see citing works
Understanding Hallucinations in Large Visual and Language Models
Ho, Z. Y., Liang, S., & Tao, D. (2026) · ACM Computing Surveys · doi:10.1145/3811409 · Cited by 3 · see citing works
UKRAG: A Unified Knowledge Graph to Enhance Retrieval Augmented Generation Performance
Alkouz, A., Al-Saleh, M. I., Alarabeyyat, A., & Bouchahma, M. (2025) · Communications in computer and information science · doi:10.1007/978-3-031-82931-4_1 · Cited by 1 · see citing works
Uniform Complexity for Text Generation
Imperial, J. M., & Madabushi, H. T. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.805 · Cited by 2 · see citing works
Predicting typeScript type annotations and definitions with machine learning
Yee, M. (2024) · doi:10.17760/d20653005 · Cited by 1 · see citing works
Document Understanding with Deep Learning Techniques
Nguyen, K.-A. L. (2024) · theses.fr (ABES) · doi:10.70675/b83ce709z5fc9z4a33za0bezd7aaad779ef1 · Cited by 1 · see citing works
CLMSM: A Multi-Task Learning Framework for Pre-training on Procedural Text
Nandy, A., Kapadnis, M. N., Goyal, P., & Ganguly, N. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.589
Topological constraints on self-organization in locally interacting systems
Sacco, F. M., Sakthivadivel, D. A. R., & Levin, M. (2026) · Philosophical Transactions of the Royal Society A Mathematical Physical and Engineering Sciences · doi:10.1098/rsta.2025.0011
Beyond the Utterance: An Empirical Study of Very Long Context Speech Recognition
Flynn, R., & Ragni, A. (2026) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2026.3658246
Majumder, S., & Sarkar, S. C. (2026) · Research Square · doi:10.21203/rs.3.rs-9986413/v1
An AI Framework for Literature Review: In Search of Sustainable Fuels
Oliveira, J. P., Mendes, A., & Ferreira, M. C. (2026) · World sustainability series · doi:10.1007/978-3-032-22076-9_40
Pretrained language models for selection of cohorts from electronic health records
Spiero, I., Damen, J. A., Hooft, L., Moons, K. G. M., & Leeuwenberg, A. M. (2026) · Informatics in Medicine Unlocked · doi:10.1016/j.imu.2026.101803
Gao, Y., Xu, N., Jin, G., Zhang, S., & Liu, A.-A. (2026) · doi:10.1109/cvprf71909.2026.00553
Alleviating Hallucinations in Large Vision-Language Models via Decoding-Time Perturbation Adaptation
Bai, J.-Q., Guo, H.-C., Liu, J.-H., Zhou, Z.-B., Yang, J., & Huang, F.-R. (2026) · 2026 IEEE/CVF Conference on Computer Vision and Pattern Recognition- FINDINGS Track (CVPRF) · doi:10.1109/cvprf71909.2026.00944
- Authors
- Sun, S., Krishna, K., Mattarella-Micke, A., & Iyyer, M.
- Affiliations
- University of Massachusetts Amherst (US)
- Venue
- Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing
- Published
- 2021
- Pages
- 807-822
- OpenAlex
- W3200409031
- Cited by
- 43
- References
- 46
- Citations by year
- 2026: 8 · 2025: 7 · 2024: 9 · 2023: 11 · 2022: 8
- FWCI
- 3.23
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1955567)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- long-range context, literary novels
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
