The DOI matches its Crossref record; 2 of the 2 references it cites were found in a registry or an index.
Not All Tokens Are What You Need for Pretraining
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Recent research indicates that token reduction has the potential to significantly improve training stability when integrated into the pre-training phase, suggesting that selective token utilization during training can lead to more robust model learning.
“While token reduction has traditionally been employed as a post-training optimization to enhance inference efficiency, recent research indicates its potential to significantly improve training stability when integrated into the pre-training phase [36, 83, 86], suggesting that selective token utilization during training can lead to more robust model learning, as shown in Fig. 2d.”
From Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodality (2025) · cites this paper as [86]
Citations
Cites
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550 · cited 3×
“1 Introduction Scaling up model parameters and dataset size has consistently elevated the next-token prediction accuracy in large language models, yielding significant advancements in artificial intelligence [Kaplan et al., 2020, Brown et al., 2020, OpenAI, 2023, Team et al., 2023].”
AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models
Zhong, W., Cui, R., Guo, Y., Liang, Y., Lu, S., Wang, Y., Saied, A., Chen, W., & Duan, N. (2023) · arXiv · doi:10.48550/arxiv.2304.06364 · Cited by 211 · cited 1×
“…we followed the lm-evaluation-harness [Gao et al., 2023] and evalute model on MMLU [Hendrycks et al., 2020], BBH [Suzgun et al., 2022], AGIEval [Zhong et al., 2023b], ARC-Easy and ARC-Challenge [Clark et al., 2018], BoolQ [Clark et al., 2019], PIQA [Bisk et al., 2020], Hellaswag [Zellers et…”
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Kong, Z., Li, Y., Zeng, F., Xin, L., Messica, S., Lin, X., Zhao, P., Kellis, M., Tang, H., & Zitnik, M. (2025) · arXiv · doi:10.48550/arxiv.2505.18227 · Cited by 51
“…to enhance inference efficiency, recent research indicates its potential to significantly improve training stability when integrated into the pre-training phase [36, 83, 86], suggesting that selective token utilization during training can lead to more robust model learning, as shown in Fig.”
Metadata
- Authors
- Lin, Z., Gou, Z., Gong, Y., Liu, X., Shen, Y., Xu, R., Lin, C., Yang, Y., Jiao, J., Duan, N., & Chen, W.
- Venue
- Advances in Neural Information Processing Systems 37
- Published
- 2024
- Pages
- 29029-29063
- ISSN
- 1049-5258 (ISSN-L)
- OpenAlex
- W4415798175
- Cited by
- 9
- References
- 2
- Citations by year
- 2026: 9
- FWCI
- 2.66
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- neural information processing systems (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- pre-training, tokenization, token selection, language models, representation learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
