CoChat
Sign Up
PreprintCited by 3Topic ModelingNatural Language Processing TechniquesMultimodal Machine Learning Applications

The DOI matches its DataCite record; its reference list is not available.

A Comprehensive Survey on Long Context Language Modeling

Open access
Liu, Jiaheng, Zhu, Dawei, Bai, Zhiqi, He, Yancheng, Liao, Huanxuan, Que, Haoran, Wang, Zekun, Zhang, Chenchen, Zhang, Ge, Zhang, Jiebin, Zhang, Yuanxing, Chen, Zhuo, Guo, Hangyu, Li, Shilong, Liu, Ziqiang, Shan, Yong, Song, Yifan, Tian, Jiayi, Wu, Wenhao, Zhou, Zhejian, Zhu, Ruijie, Feng, Junlan, Gao, Yang, He, Shizhu, Li, Zhoujun, Liu, Tianyu, Meng, Fanyu, Su, Wenbo, Tan, Yingshui, Wang, Zili, Yang, Jian, Ye, Wei, Zheng, Bo, Zhou, Wangchunshu, Huang, Wenhao, Li, Sujian, Zhang, Zhaoxiang·2025·arXiv
DOI 10.48550/arxiv.2503.17407arXiv 2503.17407
Open this paper in CoChat — free accountOpen PDF

Abstract

Efficient processing of long contexts has been a persistent pursuit in Natural Language Processing. With the growing number of long documents, dialogues, and other textual data, it is important to develop Long Context Language Models (LCLMs) that can process and analyze extensive inputs in an effective and efficient way. In this paper, we present a comprehensive survey on recent advances in long-context modeling for large language models. Our survey is structured around three key aspects: how to obtain effective and efficient LCLMs, how to train and deploy LCLMs efficiently, and how to evaluate and analyze LCLMs comprehensively. For the first aspect, we discuss data strategies, architectural designs, and workflow approaches oriented with long context processing. For the second aspect, we provide a detailed examination of the infrastructure required for LCLM training and inference. For the third aspect, we present evaluation paradigms for long-context comprehension and long-form generation, as well as behavioral analysis and mechanism interpretability of LCLMs. Beyond these three key aspects, we thoroughly explore the diverse application scenarios where existing LCLMs have been deployed and outline promising future development directions. This survey provides an up-to-date review of the literature on long-context LLMs, which we wish to serve as a valuable resource for both researchers and engineers. An associated GitHub repository collecting the latest papers and repos is available at: \href{https://github.com/LCLM-Horizon/A-Comprehensive-Survey-For-Long-Context-Language-Modeling}{\color[RGB]{175,36,67}{LCLM-Horizon}}.

Overview

Integrity signals

Registry

DataCite · 10.48550/arxiv.2503.17407

Cites

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • For long-horizon LLM agents, conditioning on the full history is impractical.

    “For long-horizon LLM agents, since the interaction history grows monotonically and suffers from 'lost-in-the-middle' degradation (Liu et al., 2024; 2025), conditioning on the full history h t is impractical.”

    From Meta-Cognitive Memory Policy Optimization for Long-Horizon LLM Agents (2026) · cites this paper as Liu et al., 2025

Citations

No citation lists for this paper

No citation lists for this paper yet — checked again daily.

Cited by 3

Metadata

Authors
Liu, J., Zhu, D., Bai, Z., He, Y., Liao, H., Que, H., Wang, Z., Zhang, C., Zhang, G., Zhang, J., Zhang, Y., Chen, Z., Guo, H., Li, S., Liu, Z., Shan, Y., Song, Y., Tian, J., Wu, W., Zhou, Z., Zhu, R., Feng, J., Gao, Y., He, S., Li, Z., Liu, T., Meng, F., Su, W., Tan, Y., Wang, Z., Yang, J., Ye, W., Zheng, B., Zhou, W., Huang, W., Li, S., & Zhang, Z.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2503.17407
arXiv
2503.17407
OpenAlex
W6903651657
Cited by
3
References
0
Citations by year
2026: 2 · 2025: 1
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
long-context language modeling, long-form text generation, large language models, long-range dependencies

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review