Physics of language models: Part 3.3, knowledge capacity scaling laws
Open accessAbstract
Scaling laws describe the relationship between the size of language models and their capabilities. Unlike prior studies that evaluate a model's capability via loss or benchmarks, we estimate information-theoretically the number of knowledge bits a model stores. We focus on factual knowledge represented as tuples, such as (USA, capital, Washington D.C.) from a Wikipedia page. Through multiple controlled datasets, we establish that language models can and only can store 2 bits of knowledge per parameter, even when quantized to int8, and such knowledge can be flexibly extracted for downstream applications. Consequently, a 7B model can store 14B bits of knowledge, surpassing the English Wikipedia and textbooks combined based on our estimation. More broadly, we present 12 results on how (1) training duration, (2) model architecture, (3) quantization, (4) sparsity constraints such as MoE, and (5) data signal-to-noise ratio affect a model's knowledge storage capacity. Notable insights include: * The GPT-2 architecture, with rotary embedding, matches or even surpasses LLaMA/Mistral architectures in knowledge storage, particularly over shorter training durations. This arises because LLaMA/Mistral uses GatedMLP, which is less stable and harder to train. * Prepending training data with domain names (e.g., wikipedia.org) significantly increases a model's knowledge capacity. Language models can autonomously identify and prioritize domains rich in knowledge, optimizing their storage capacity.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
One single side memory has limited knowledge capacity.
“One single side memory has its limited knowledge capacity [43].”
From WISE: Rethinking the Knowledge Memory for Lifelong Model Editing of Large Language Models (2024) · cites this paper as [43]
The Abstract (p. 1) reports a knowledge-capacity limit for language models of 2 bits per parameter. The paper addresses knowledge stored in model parameters, not the capacity of a separate “side memory,” so that part of the claim is unsupported. (LLM, conf=0.84)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
LoRA: Low-Rank Adaptation of Large Language Models
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2021) · arXiv · doi:10.48550/arxiv.2106.09685 · Cited by 2,523 · see citing works
Self-citationNatural Questions: A benchmark for question answering research
Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., et al. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00276 · Cited by 2,069 · see citing works
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
Fedus, W., Zoph, B., & Shazeer, N. (2021) · arXiv · doi:10.48550/arxiv.2101.03961 · Cited by 1,005 · see citing works
Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., & Dean, J. (2017) · arXiv · doi:10.48550/arxiv.1701.06538 · Cited by 933 · see citing works
A Convergence Theory for Deep Learning via Over-ParameterizationPreprint
Allen-Zhu, Z., Li, Y., & Song, Z. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.03962 · Cited by 847 · see citing works
Self-citationLearning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured DataPreprint
Li, Y., & Liang, Y. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1808.01204 · Cited by 414 · see citing works
Self-citationLearning and Generalization in Overparameterized Neural Networks, Going Beyond Two LayersPreprint
Allen-Zhu, Z., Li, Y., & Liang, Y. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.04918 · Cited by 154 · see citing works
Self-citationGPT-NeoX-20B: An Open-Source Autoregressive Language Model
Black, S., Biderman, S., Hallahan, E., Anthony, Q., Gao, L., Golding, L., He, H., Leahy, C., McDonell, K., Phang, J., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.06745 · Cited by 67 · see citing works
Revisiting neural scaling laws in language and vision
Alabdulmohsin, I., Neyshabur, B., & Zhai, X. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1620 · Cited by 41 · see citing works
Flagged first, then by citations
Research works printed but not linked: 43
- 0616, N=10000000 N=5000000 N=2000000 N=1000000 N=500000 N=200000 N=100000 N=50000 N=20000 N=10000
- 25 bit / param N=10000000 N=5000000 N=2000000 N=1000000 N=500000 N=200000 N=100000 N=50000 N=20000 N=10000 (a) Same Figure, 1, 1000
- 25 bit / param N=10000000 N=5000000 N=2000000 N=1000000 N=500000 N=200000 N=100000 N=50000 N=20000 N=10000
- 25 bit / param N=1000000
- N=500000 N=200000 N=100000 N=50000 N=20000 N=10000 (c) Figure 1(a)
- 25 bit / param N=20000000 N=10000000 N=5000000 N=2000000 N=1000000 N=500000 N=200000 N=100000 N=50000 (d) Same Figure 1(b), 100
- 25 bit / param N=20000000 N=10000000 N=5000000 N=2000000 N=1000000 N=500000 N=200000 N=100000 N=50000
- k-D 10 -C 1-K5 -L1 -T4 0k N5 00 k-D 10 -C 1-K1 0-L1 -T4 0k N5 00 k-D 10 -C 1-K2 0-L1 -T4 0k N2 00 k-D 10 -C 1-K
- 25 bit / param N=10000000 N=5000000 N=2000000 N=1000000 N=500000 N=200000 N=100000 N=50000 N=20000 N=10000
- 25 bit / param N=1000000
- N=500000 N=200000 N=100000 N=50000
- 25 bit / param N=1000000
- N=500000 N=200000 N=100000 N=50000
- Q N +jD+D = [T L ]
- Recall that each Q i is independently and uniformly generated at random from Q i . We now present an alternative method for generating the training dataset
- N Construct, N ) as follows: Let n 1 be the Q 1 -th name from N 0 ; for i > 1, let n i be the Q i -th name from N 0 \ {n 1
- let a be the a ? -th attribute in A. Construct D a = (w 1 , . . . , w D ) as follows: Let w 1 be the Q N +(a ? -1)D+1 -th element in T L ; for i > 1, let w i be the Q N +(a ? -1)D+i -th element in T L \ {w 1
- Zeyuan Allen, 2023, Physics of Language Models: Part 1, Context-Free Grammar
- Zeyuan Allen, 2023, Physics of Language Models: Part 3.1, Knowledge Storage and Extraction
- Zeyuan Allen, 2023, Physics of Language Models: Part 3.2, Knowledge Manipulation
- Zeyuan Allen-Zhu, 2019, A convergence theory for deep learning via overparameterization, International conference on machine learning, 242
- S�bastien Bubeck, 2023, Sparks of artificial general intelligence: Early experiments with gpt-4
- Elias Frantar, 2022, GPTQ: Accurate post-training compression for generative pretrained transformers
- Olga Golovneva, 2024, Reverse training to nurse the reversal curse
- Suriya Gunasekar, 2023, Textbooks are all you need
- Tom Henighan, 2020, Scaling laws for autoregressive generative modeling
- Danny Hernandez, 2021, Scaling laws for transfer
- Joel Hestness, 2017, Deep learning scaling is predictable, empirically
- Jordan Hoffmann, 2022, Training computeoptimal large language models
- Changho Hwang, 2022, Tutel: Adaptive mixture-of-experts at scale
- Alexandre Albert Q Jiang, 2023, Mistral 7b
- Mandar Joshi, 2017, Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension
- Jared Kaplan, 2020, Scaling laws for neural language models
- Yuanzhi Li, 2023, Textbooks are all you need ii: phi-1.5 technical report
- Niklas Muennighoff, 2023, Aleksandra Piktus, Nouamane Tazi, Sampo Pyysalo, Thomas Wolf, and Colin Raffel. Scaling data-constrained language models
- Alec Radford, 2019, Language models are unsupervised multitask learners, OpenAI blog, 1
- Jonathan S Rosenfeld, 2021, Scaling laws for deep learning
- Jonathan S Rosenfeld, 2019, A constructive prediction of the generalization error across scales
- Noam Shazeer, 2020, Glu variants improve transformer
- Jianlin Su, 2021, Roformer: Enhanced transformer with rotary position embedding
- Hugo Touvron, 2023, Open and efficient foundation language models
- Hugo Touvron, 2023, Llama 2: Open foundation and fine-tuned chat models
- Dingli Yu, 2023, Skillmix: A flexible and expandable family of evaluations for ai models
Cited by
Sorted by citations
State of the Art and Future Directions of Small Language Models: A Systematic Review
Corradini, F., Leonesi, M., & Piangerelli, M. (2025) · Big Data and Cognitive Computing · doi:10.3390/bdcc9070189 · Cited by 28 · see citing works
The spatial complexity of optical computing: toward space-efficient design
Li, Y., & Monticone, F. (2025) · Nature Communications · doi:10.1038/s41467-025-63453-8 · Cited by 12 · see citing works
Self-citationPhysics of Language Models: Part 1, Learning Hierarchical Language StructuresPreprint
Allen-Zhu, Z., & Li, Y. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5250639 · Cited by 6 · see citing works
Self-citationWISE: Rethinking the Knowledge Memory for Lifelong Model Editing of Large Language Models
Wang, P., Li, Z., Zhang, N., Xu, Z., Yao, Y., Jiang, Y., Xie, P., Huang, F., & Chen, H. (2024) · arXiv · doi:10.48550/arxiv.2405.14768 · Cited by 19 · see citing works
Metadata Conditioning Accelerates Language Model Pre-trainingPreprint
Gao, T., Wettig, A., He, L., Dong, Y., Malladi, S., & Chen, D. (2025) · Qeios · doi:10.32388/d226qh · Cited by 5 · see citing works
Detecting and quantifying overparametrization in RNA language models with REDIALPreprint
Teng, D., Qiu, Y.-R., Sakthivel, G., Aranganathan, A., Herron, L., & Tiwary, P. (2026) · bioRxiv · doi:10.64898/2026.05.11.724344 · Cited by 2 · see citing works
MARS6: A Small and Robust Hierarchical-Codec Text-to-Speech Model
Baas, M., Scholtz, P., Mehta, A., Dyson, E., Prakash, A., & Kamper, H. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10890838 · Cited by 1 · see citing works
Pan, X., Wang, R., Lin, W., Shi, S., & Chu, X. (2025) · IEEE International Conference on Distributed Computing Systems (ICDCS) · doi:10.1109/icdcs63083.2025.00063 · Cited by 1 · see citing works
Ringwald, C., Gandon, F. L., Zucker, C. F., Michel, F., & Akl, H. A. (2025) · International Conference on Knowledge Capture · doi:10.1145/3731443.3771342 · Cited by 1 · see citing works
Staying VIGILant: Mitigating Visual Laziness via Counterfactual Visual Alignment in MLLMs
Xiao, X., Liu, C., Liao, C.-T., Zhang, Y., Lan, Q., Wei, Y., Zhao, L., Wang, J., Gu, J., Ye, M., et al. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37232-1_16 · Cited by 3 · see citing works
Understanding large language models demands distinguishing human projection from machine cognition
Li, L., Teng, Y., Wang, Y., & Hu, X. (2026) · Communications Psychology · doi:10.1038/s44271-026-00508-6
The Information Theory of Self-Organization Phenomena in Thermal SystemsPreprint
Liu, H., & Wu, Z. (2024) · Qeios · doi:10.32388/mts9l8
ASMem: Anchor sparse memory for multi-domain knowledge editing of large language models
Zheng, G., Wang, Z., Zhao, Y., He, T., Wang, X., Wang, H., Zhao, T., & Zong, C. (2026) · Neural Networks · doi:10.1016/j.neunet.2026.109230
Evaluating the Critical Thinking of Large Language Models: Insights and Limitations
Jiang, L., Cui, Y. D., Liu, R., & Luo, F. (2025) · Journal of Pacific Rim Psychology · doi:10.1177/18344909251406111
Исследование раннего и позднего коллапса языковых моделей в медицинских приложениях
Боброва, Е. В., Сергеевич, З. К., Константинович, С. Д., Витальевич, Х. Д., & Владимирович, Д. Е. (2025) · International journal of open information technologies · doi:10.25559/injoit.2307-8162.13.202508.51-59
Wang, Y., Wan, C., Tian, X., Liu, X., & Chen, Y. (2025) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata66926.2025.11401489
On Memorization and Generalization in Compact Transformers
Härmä, A., Al-Saeedi, A., Changalidis, A., Verșebeniuc, D., Pietrasik, M., & Wilbik, A. (2026) · Electronics · doi:10.3390/electronics15091847
Akyol, E. (2026) · IEEE Journal on Selected Areas in Information Theory · doi:10.1109/jsait.2026.3699650
- Authors
- Allen-Zhu, Z., & Li, Y.
- Affiliations
- Meta (United States) (US)
- Mohamed bin Zayed University of Artificial Intelligence (AE)
- Corresponding author
- Allen-Zhu, Zeyuan
- Venue
- SSRN Electronic Journal
- Publisher
- Elsevier BV
- Published
- 2025
- ISSN
- 1556-5068 (ISSN-L)
- OpenAlex
- W4410356599
- Cited by
- 17
- References
- 9
- Citations by year
- 2026: 6 · 2025: 10 · 2024: 1
- Type
- posted content
- Language
- en
- Open access
- green
- doi.org
- Locations
- SSRN Electronic Journal (acceptedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- scaling laws, information-theoretic estimation, model quantization, training duration, Wikipedia data, knowledge extraction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
