CoChat
Sign Up
PreprintCited by 5Topic ModelingInformation Retrieval and Search BehaviorNatural Language Processing Techniques

Meta-Chunking: Learning Text Segmentation and Semantic Completion via Logical Perception

Open access
Zhao, Jihao, Ji, Zhiyuan, Feng, Yuchen, Qi, Pengnian, Niu, Simin, Tang, Bo, Xiong, Feiyu, Li, Zhiyu·2024·arXiv
DOI 10.48550/arxiv.2410.12788arXiv 2410.12788
Open this paper in CoChat — free accountOpen PDF

Abstract

While Retrieval-Augmented Generation (RAG) has emerged as a promising paradigm for boosting large language models (LLMs) in knowledge-intensive tasks, it often overlooks the crucial aspect of text chunking within its workflow. This paper proposes the Meta-Chunking framework, which specifically enhances chunking quality through a dual strategy that identifies optimal segmentation points and preserves global information. Initially, breaking limitations of similarity-based chunking, we design two adaptive chunking techniques based on uncertainty, namely Perplexity Chunking and Margin Sampling Chunking, by utilizing the logical perception capabilities of LLMs. Given the inherent complexity across different texts, we integrate meta-chunk with dynamic merging, striking a balance between fine-grained and coarse-grained text chunking. Furthermore, we establish the global information compensation mechanism, encompassing a two-stage hierarchical summary generation process and a three-stage text chunk rewriting procedure focused on missing reflection, refinement, and completion. These components collectively strengthen the semantic integrity and contextual coherence of chunks. Extensive experiments demonstrate that Meta-Chunking effectively addresses challenges of the chunking task within the RAG system, providing LLMs with more logically coherent text chunks. Additionally, our methodology validates the feasibility of implementing high-quality chunking tasks with smaller-scale models, thereby eliminating the reliance on robust instruction-following capabilities.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2410.12788

Cites

0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Perplexity-based grouping has shown effectiveness in natural language segmentation but remains under-explored in code.

    “While perplexity-based grouping has shown effectiveness in natural language segmentation, it remains under-explored in code.”

    From LongCodeZip: Compress Long Context for Code Language Models (2025) · cites this paper as [32]

    Section 3.1.1 (p. 3) describes grouping sentences into text chunks using perplexity, and Section 5.1 (p. 7) reports that PPL Chunking balances performance and processing time. The paper does not address whether perplexity-based grouping is under-explored in code; its statement that chunking has received insufficient investigation concerns text chunking in RAG, not code. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • Within a semantically coherent region, perplexity tends to decrease as context accumulates.

    “Within a semantically coherent region, perplexity tends to decrease as context accumulates [32].”

    From LongCodeZip: Compress Long Context for Code Language Models (2025) · cites this paper as [32]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 41

  1. [1][1] H. He, H. Zhang, and D. Roth, 'Rethinking with retrieval: Faithful large language model inference,' arXiv preprint arXiv:2301.00303 , 2022. - · arXiv
  2. [2][2] Y. Chen, Q. Fu, Y. Yuan, Z. Wen, G. Fan, D. Liu, D. Zhang, Z. Li, and Y. Xiao, 'Hallucination detection: Robustly discerning reliable answers in large language models,' in Proceedings of the 32nd ACM International Conference on Information and Knowledge Management , 2023, pp. 245-255. -
  3. [3][3] G. Zuccon, B. Koopman, and R. Shaik, 'Chatgpt hallucinates when attributing answers,' in Proceedings of the Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region , 2023, pp. 46-51. -
  4. [4][4] X. Liang, S. Song, Z. Zheng, H. Wang, Q. Yu, X. Li, R.-H. Li, F. Xiong, and Z. Li, 'Internal consistency and self-feedback in large language models: A survey,' arXiv preprint arXiv:2407.14507 , 2024. - · arXiv
  5. [5][5] X. Li, S. Chan, X. Zhu, Y. Pei, Z. Ma, X. Liu, and S. Shah, 'Are chatgpt and gpt-4 generalpurpose solvers for financial text analytics? a study on several typical tasks,' arXiv preprint arXiv:2305.05862 , 2023. - · arXiv
  6. [7][7] A. Lazaridou, E. Gribovskaya, W. Stokowiec, and N. Grigorev, 'Internet-augmented language models through few-shot prompting for open-domain question answering,' arXiv preprint arXiv:2203.05115 , 2022. - · arXiv
  7. [8][8] H. Li, Y. Su, D. Cai, Y. Wang, and L. Liu, 'A survey on retrieval-augmented text generation,' arXiv preprint arXiv:2202.01110 , 2022. - · arXiv
  8. [9][9] C.-H. Tan, J.-C. Gu, C. Tao, Z.-H. Ling, C. Xu, H. Hu, X. Geng, and D. Jiang, 'Tegtok: Augmenting text generation via task-specific and open-world knowledge,' arXiv preprint arXiv:2203.08517 , 2022. - · arXiv
  9. [10][10] W. Lin, R. Blloshmi, B. Byrne, A. de Gispert, and G. Iglesias, 'Li-rage: Late interaction retrieval augmented generation with explicit signals for open-domain table question answering,' in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) , 2023, pp. 1557-1566. -
  10. [11][11] S. Xu, L. Pang, H. Shen, and X. Cheng, 'Berm: Training the balanced and extractable representation for matching to improve generalization ability of dense retrieval,' arXiv preprint arXiv:2305.11052 , 2023. - · arXiv
  11. [12][12] W. Su, Y. Tang, Q. Ai, Z. Wu, and Y. Liu, 'Dragin: Dynamic retrieval augmented generation based on the real-time information needs of large language models,' arXiv preprint arXiv:2403.10081 , 2024. - · arXiv
  12. [13][13] M. Besta, A. Kubicek, R. Niggli, R. Gerstenberger, L. Weitzendorf, M. Chi, P. Iff, J. Gajda, P. Nyczyk, J. Müller et al. , 'Multi-head rag: Solving multi-aspect problems with llms,' arXiv preprint arXiv:2406.05085 , 2024. - · arXiv
  13. [14][14] G. Sidiropoulos and E. Kanoulas, 'Analysing the robustness of dual encoders for dense retrieval against misspellings,' in Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval , 2022, pp. 2132-2136. -
  14. [15][15] Z. Zhuang, Z. Zhang, S. Cheng, F. Yang, J. Liu, S. Huang, Q. Lin, S. Rajmohan, D. Zhang, and Q. Zhang, 'Efficientrag: Efficient retriever for multi-hop question answering,' arXiv preprint arXiv:2408.04259 , 2024. - · arXiv
  15. [16][16] Y. Kim, H. J. Kim, C. Park, C. Park, H. Cho, J. Kim, K. M. Yoo, S.-g. Lee, and T. Kim, 'Adaptive contrastive decoding in retrieval-augmented generation for handling noisy contexts,' arXiv preprint arXiv:2408.01084 , 2024. - · arXiv
  16. [17][17] Q. Zhang, Q. Chen, Y. Li, J. Liu, and W. Wang, 'Sequence model with self-adaptive sliding window for efficient spoken document segmentation,' in 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) . IEEE, 2021, pp. 411-418. -
  17. [19][19] Y. Lyu, Z. Li, S. Niu, F. Xiong, B. Tang, W. Wang, H. Wu, H. Liu, T. Xu, and E. Chen, 'Crudrag: A comprehensive chinese benchmark for retrieval-augmented generation of large language models,' arXiv preprint arXiv:2401.17043 , 2024. - · arXiv
  18. [20][20] A. V. Duarte, J. Marques, M. Graça, M. Freire, L. Li, and A. L. Oliveira, 'Lumberchunker: Long-form narrative document segmentation,' arXiv preprint arXiv:2406.17526 , 2024. - · arXiv
  19. [21][21] K. Guu, K. Lee, Z. Tung, P. Pasupat, and M. Chang, 'Retrieval augmented language model pre-training,' in International conference on machine learning . PMLR, 2020, pp. 3929-3938. -
  20. [22][22] P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W.-t. Yih, T. Rocktäschel et al. , 'Retrieval-augmented generation for knowledge-intensive nlp tasks,' Advances in Neural Information Processing Systems , vol. 33, pp. 9459-9474, 2020. -
  21. [23][23] O. Ram, Y. Levine, I. Dalmedigos, D. Muhlgay, A. Shashua, K. Leyton-Brown, and Y. Shoham, 'In-context retrieval-augmented language models,' Transactions of the Association for Computational Linguistics , vol. 11, pp. 1316-1331, 2023. -
  22. [24][24] W. Yu, H. Zhang, X. Pan, K. Ma, H. Wang, and D. Yu, 'Chain-of-note: Enhancing robustness in retrieval-augmented language models,' arXiv preprint arXiv:2311.09210 , 2023. - · arXiv
  23. [25][25] Y. Gao, Y. Xiong, X. Gao, K. Jia, J. Pan, Y. Bi, Y. Dai, J. Sun, and H. Wang, 'Retrievalaugmented generation for large language models: A survey,' arXiv preprint arXiv:2312.10997 , 2023. - · arXiv
  24. [26][26] T. Chen, H. Wang, S. Chen, W. Yu, K. Ma, X. Zhao, D. Yu, and H. Zhang, 'Dense x retrieval: What retrieval granularity should we use?' arXiv preprint arXiv:2312.06648 , 2023. - · arXiv
  25. [27][27] R. Zhang, H. Zhang, and Z. Zheng, 'Vl-uncertainty: Detecting hallucination in large visionlanguage model via uncertainty estimation,' arXiv preprint arXiv:2411.11919 , 2024. - · arXiv
  26. [28][28] Y. Li, R. Qiang, L. Moukheiber, and C. Zhang, 'Language model uncertainty quantification with attention chain,' arXiv preprint arXiv:2503.19168 , 2025. - · arXiv
  27. [29][29] L. Da, X. Liu, J. Dai, L. Cheng, Y. Wang, and H. Wei, 'Understanding the uncertainty of llm explanations: A perspective based on reasoning topology,' arXiv preprint arXiv:2502.17026 , 2025. - · arXiv
  28. [30][30] Z. Atf, S. A. A. Safavi-Naini, P. R. Lewis, A. Mahjoubfar, N. Naderi, T. R. Savage, and A. Soroush, 'The challenge of uncertainty quantification of large language models in medicine,' arXiv preprint arXiv:2504.05278 , 2025. - · arXiv
  29. [31][31] S. Farquhar, J. Kossen, L. Kuhn, and Y. Gal, 'Detecting hallucinations in large language models using semantic entropy,' Nature , vol. 630, no. 8017, pp. 625-630, 2024. -
  30. [32][32] X. Liu, T. Chen, L. Da, C. Chen, Z. Lin, and H. Wei, 'Uncertainty quantification and confidence calibration in large language models: A survey,' arXiv preprint arXiv:2503.15850 , 2025. - · arXiv
  31. [33][33] Y. Abbasi Yadkori, I. Kuzborskij, A. György, and C. Szepesvari, 'To believe or not to believe your llm: Iterative prompting for estimating epistemic uncertainty,' Advances in Neural Information Processing Systems , vol. 37, pp. 58 077-58 117, 2024. -
  32. [34][34] R. Qu, R. Tu, and F. Bao, 'Is semantic chunking worth the computational cost?' arXiv preprint arXiv:2410.13070 , 2024. - · arXiv
  33. [35][35] C. E. Shannon, 'Prediction and entropy of printed english,' Bell system technical journal , vol. 30, no. 1, pp. 50-64, 1951. -
  34. [36][36] Y. Bai, X. Lv, J. Zhang, H. Lyu, J. Tang, Z. Huang, Z. Du, X. Liu, A. Zeng, L. Hou et al. , 'Longbench: A bilingual, multitask benchmark for long context understanding,' arXiv preprint arXiv:2308.14508 , 2023. - · arXiv
  35. [37][37] S. Xiao, Z. Liu, P. Zhang, and N. Muennighof, 'C-pack: packaged resources to advance general chinese embedding. 2023,' arXiv preprint arXiv:2309.07597 , 2023. - · arXiv
  36. [38][38] J. He, M. Rungta, D. Koleczek, A. Sekhon, F. X. Wang, and S. Hasan, 'Does prompt formatting have any impact on llm performance?' arXiv preprint arXiv:2411.10541 , 2024. - · arXiv
  37. [39][39] Y.-C. Chang, M.-S. Huang, Y.-H. Huang, and Y.-H. Lin, 'The influence of prompt engineering on large language models for protein-protein interaction identification in biomedical literature,' Scientific Reports , vol. 15, no. 1, p. 15493, 2025. -
  38. [40][40] S. Srivastava and Z. Yao, 'Revisiting prompt optimization with large reasoning models-a case study on event extraction,' arXiv preprint arXiv:2504.07357 , 2025. - · arXiv
  39. [41][41] C. Huyen, 'Evaluation metrics for language modeling,' The Gradient , vol. 40, 2019. -
  40. [42][42] S. Dragomir and C. Goh, 'Some bounds on entropy measures in information theory,' Applied Mathematics Letters , vol. 10, no. 3, pp. 23-28, 1997. -
  41. [43][43] Y. Tang and Y. Yang, 'Multihop-rag: Benchmarking retrieval-augmented generation for multihop queries (2024),' arXiv preprint arXiv:2401.15391 . - · arXiv

Cited by

Sorted by citations

Related

Authors
Zhao, J., Ji, Z., Feng, Y., Qi, P., Niu, S., Tang, B., Xiong, F., & Li, Z.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2410.12788
arXiv
2410.12788
OpenAlex
W4403583523
Cited by
1
References
0
Citations by year
2026: 1
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
text chunking, retrieval-augmented generation, hierarchical summarization, large language models

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review