The DOI matches its DataCite record; 1 of the 16 references it cites were found in a registry or an index, and 15 are not checked yet.
DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
Open accessAbstract
We present DeepSeek-V2, a strong Mixture-of-Experts (MoE) language model characterized by economical training and efficient inference. It comprises 236B total parameters, of which 21B are activated for each token, and supports a context length of 128K tokens. DeepSeek-V2 adopts innovative architectures including Multi-head Latent Attention (MLA) and DeepSeekMoE. MLA guarantees efficient inference through significantly compressing the Key-Value (KV) cache into a latent vector, while DeepSeekMoE enables training strong models at an economical cost through sparse computation. Compared with DeepSeek 67B, DeepSeek-V2 achieves significantly stronger performance, and meanwhile saves 42.5% of training costs, reduces the KV cache by 93.3%, and boosts the maximum generation throughput to 5.76 times. We pretrain DeepSeek-V2 on a high-quality and multi-source corpus consisting of 8.1T tokens, and further perform Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) to fully unlock its potential. Evaluation results show that, even with only 21B activated parameters, DeepSeek-V2 and its chat versions still achieve top-tier performance among open-source models.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Such an auxiliary loss can be applied at different granularities (e.g., at mini-batch, device, and sequence levels [40,67]), thereby improving communication efficiency.”
From A Survey of Large Language Models (2026) · cites this paper as [67]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, while the open-source model ecosystem has advanced rapidly on the second axis, it has progressed slowly on the first: many recent models remain within or slightly above the 1T-class parameter regime [145,29,135,120].”
From Kimi K3: Open Frontier Intelligence (2026) · cites this paper as [29]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“To reduce communication when distributing experts across different devices using expert parallelism, device-limited routing has been introduced [67], which first selects a small number of devices and then applies topselection only to the experts on those devices.”
From A Survey of Large Language Models (2026) · cites this paper as [67]
The cited source says what the claim attributes to it.
In the citing paper
“At the model level, KV cache reduction can be achieved through architectural modifications, such as layer-wise KV sharing [298] and multi-head latent attention (MLA) [67,299].”
From A Survey of Large Language Models (2026) · cites this paper as [67]
The cited source says what the claim attributes to it.
In the citing paper
“While this effect aligns with explicit compression methods such as MLA [66] and ASVD [42], the approach described herein achieves this via unified factorization within attention layers, thereby reducing both weight and activation memory simultaneously.”
From LittleBit: Ultra Low-Bit Quantization via Latent Factorization (2025) · cites this paper as [66]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 32
- J. Ainslie, J. Lee-Thorp, M. de Jong, Y. Zemlyanskiy, F. Lebrón, and S. Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. arXiv preprint arXiv:2305.13245, 2023. · arXiv
- J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021. · arXiv
- J. Bai, S. Bai, Y. Chu, Z. Cui, K. Dang, X. Deng, Y. Fan, W. Ge, Y. Han, F. Huang, B. Hui, L. Ji, M. Li, J. Lin, R. Lin, D. Liu, G. Liu, C. Lu, K. Lu, J. Ma, R. Men, X. Ren, X. Ren, C. Tan, S. Tan, J. Tu, P. Wang, S. Wang, W. Wang, S. Wu, B. Xu, J. Xu, A. Yang, H. Yang, J. Yang, S. Yang, Y. Yao, B. Yu, H. Yuan, Z. Yuan, J. Zhang, X. Zhang, Y. Zhang, Z. Zhang, C. Zhou, J. Zhou, X. Zhou, and T. Zhu. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023. · arXiv
- M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba. Evaluating large language models trained on code. CoRR, abs/2107.03374, 2021. URL https://arxiv.org/abs/2107.03374. · arXiv
- P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have solved question answering? try arc, the AI2 reasoning challenge. CoRR, abs/1803.05457, 2018. URL http://arxiv.org/abs/1803.05457. · arXiv
- K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. · arXiv
- Y. Dubois, B. Galambosi, P. Liang, and T. B. Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475, 2024. · arXiv
- W. Fedus, B. Zoph, and N. Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. CoRR, abs/2101.03961, 2021. URL https://arxiv.org/ abs/2101.03961. · arXiv
- L. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, et al. The Pile: An 800GB dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020. · arXiv
- D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020. · arXiv
- D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt. Mea- suring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021. · arXiv
- S. Hu, Y. Tu, X. Han, C. He, G. Cui, X. Long, Z. Zheng, Y. Fang, Y. Huang, W. Zhao, et al. Minicpm: Unveiling the potential of small language models with scalable training strategies. arXiv preprint arXiv:2404.06395, 2024. · arXiv
- Y. Huang, Y. Bai, Z. Zhu, J. Zhang, J. Zhang, T. Su, J. Liu, C. Lv, Y. Zhang, J. Lei, et al. C-Eval: A multi-level multi-discipline chinese evaluation suite for foundation models. arXiv preprint arXiv:2305.08322, 2023. · arXiv
- N. Jain, K. Han, A. Gu, W.-D. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974, 2024. · arXiv
- W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
- D. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. In 9th International Conference on Learning Representations, ICLR 2021. OpenReview.net, 2021. URL https://openreview.net/forum?id=qrwe7XHTmYb.
- H. Li, Y. Zhang, F. Koto, Y. Yang, H. Zhao, Y. Gong, N. Duan, and T. Baldwin. CMMLU: Measur- ing massive multitask language understanding in Chinese. arXiv preprint arXiv:2306.09212, 2023. · arXiv
- I. Loshchilov and F. Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. · arXiv
- OpenAI. GPT4 technical report. arXiv preprint arXiv:2303.08774, 2023. · arXiv
- L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
- B. Peng, J. Quesnelle, H. Fan, and E. Shippole. Yarn: Efficient context window extension of large language models. arXiv preprint arXiv:2309.00071, 2023. · arXiv
- P. Qi, X. Wan, G. Huang, and M. Lin. Zero bubble pipeline parallelism. arXiv preprint arXiv:2401.10241, 2023. · arXiv
- S. Rajbhandari, J. Rasley, O. Ruwase, and Y. He. Zero: Memory optimizations toward training tril- lion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1-16. IEEE, 2020.
- C. Riquelme, J. Puigcerver, B. Mustafa, M. Neumann, R. Jenatton, A. S. Pinto, D. Keysers, and N. Houlsby. Scaling vision with sparse mixture of experts. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, pages 8583-8595, 2021. URL https://proceedings.neurips.cc/paper /2021/hash/48237d9f2dea8c74c2a72126cf63d933-Abstract.html.
- Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, M. Zhang, Y. Li, Y. Wu, and D. Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. · arXiv
- N. Shazeer. Fast transformer decoding: One write-head is all you need. CoRR, abs/1911.02150, 2019. URL http://arxiv.org/abs/1911.02150. · arXiv
- N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. V. Le, G. E. Hinton, and J. Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In 5th International Conference on Learning Representations, ICLR 2017. OpenReview.net, 2017. URL https: //openreview.net/forum?id=B1ckMDqlg.
- M. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowdhery, Q. V. Le, E. H. Chi, D. Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261, 2022. · arXiv
- A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polo- sukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
- A. Young, B. Chen, C. Li, C. Huang, G. Zhang, G. Zhang, H. Li, J. Zhu, J. Chen, J. Chang, et al. Yi: Open foundation models by 01. ai. arXiv preprint arXiv:2403.04652, 2024. · arXiv
- C. Zhou, P. Liu, P. Xu, S. Iyer, J. Sun, Y. Mao, X. Ma, A. Efrat, P. Yu, L. Yu, et al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36, 2024.
- J. Zhou, T. Lu, S. Mishra, S. Brahma, S. Basu, Y. Luan, D. Zhou, and L. Hou. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911, 2023. · arXiv
Cited by
Sorted by citations
Items 101–138 of 138
Ai, Q., Zhan, J., & Liu, Y. (2025) · arXiv · doi:10.48550/arxiv.2501.02842
“Recently, DeepSeek-AI [20] introduced multi-head latent attention, which compresses keys and values into a single latent space, thereby reducing the key-value cache while maintaining robust representational capacity.”
VarRate: Training-Free Variable-Rate KV Cache Compression for Long-Context LLMs
Esmat, S., Shah, D., & Jannesari, A. (2026) · arXiv · doi:10.48550/arxiv.2607.15498
RCR-Router: Efficient Role-Aware Context Routing for Multi-Agent LLM Systems with Structured Memory
Liu, J., Kong, Z., Yang, C., Yang, F., Li, T., Dong, P., Nanjekye, J., Tang, H., Yuan, G., Niu, W., et al. (2025) · arXiv · doi:10.48550/arxiv.2508.04903
“• Backend-Agnostic Compatibility: The modular design is compatible with diverse LLM backends (e.g., OpenAI, DeepSeek, ChatGLM), as long as agents operate under a consistent prompt-response protocol.”
InfluentialConditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models
Cheng, X., Zeng, W., Dai, D., Chen, Q., Wang, B., Xie, Z., Huang, K., Yu, X., Hao, Z., Zhang, H., et al. (2026) · Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2026.acl-long.226 · Cited by 3
Self-citationOjaKV: Context-Aware Online Low-Rank KV Cache Compression
Zhu, Y., Yang, D. H., Amiri, M. M., Murugesan, K., Pedapati, T., & Chen, P.-Y. (2025) · arXiv · doi:10.48550/arxiv.2509.21623
Self-citationHow to Tame Pre-Trained Transformers for Authorship VerificationPreprint
Li, M. Q., Fung, B. C. M., Huang, S.-C., & Fachkha, C. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4889071
Self-citationSHA: sparse adaptive head of attention
Haoyang, Y. (2025) · International Conference on Signal Processing and Computer Science (SPCS) · doi:10.1117/12.3053346
Li, W., Li, Y., Shao, Y., & Bi, M. (2025) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3725733
Self-citationStructure-Guided and Semantics-Enhanced Collaborative Code Generation
Yang, D., Zhang, X., & Cao, Y. (2025) · International Conference on Advanced Electronic Materials, Computers and Software Engineering (AEMCSE) · doi:10.1109/aemcse65292.2025.11042693 · Cited by 1
Self-citationFine-Tuning OpenAI GPT Chatbot in Western Saudi Dialect: A Case Study of Taibah University
Alhujaili, M., & Abdulrahman, R. (2025) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2025.0160632
From Screen to Scene: An In-Depth Forensic Analysis on DeepseekPreprint
Gong, Y., Tyagi, S., Mahindra, V., & Karabiyik, Ü. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5362681
Evidence-Augmented Generative Explanation for Health Rumor Detection
Tang, S., Zhong, Q., Li, P., & Zhu, Q. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-0014-7_44
Self-citationEnhancing Prediction Robustness of Latent Diffusion Models through World Models
Na, Y., Park, R. Y., Seo, S., & Kwon, T. (2025) · Journal of the Korea Computer Graphics Society · doi:10.15701/kcgs.2025.31.3.21
A simple yet effective feature completion model for relation-level learning in missing data
Chen, J., Zhang, L., Huang, Z., Liu, H., Wu, Y., Wang, H., Liu, W., & Leung, V. C. M. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103528
Self-citationPohyer, V., de Margerie‐Mellon, C., Perronne, L., Duron, L., Thibault, C., Oudard, S. M., Fournier, L. S., & Rance, B. (2025) · Research Square · doi:10.21203/rs.3.rs-7630430/v1 · Cited by 1
More In Less: Dynamic Sparse Attention andMixture-of-Experts for Iterative Vision ReasoningPreprint
Umar, A., Ahmad, I., & Ahmed, Z. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5626143
TALE: Token-Adaptive Low-Rank KVCache Approximation with Reconstruction Elimination
Lee, J., Hwang, S.-W., Qiao, A., Campos, D., Yao, Z., & He, Y. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.39
Self-citationYu, G., Bi, X., Zhao, Q., & Wang, J. (2026) · Bioinformatics · doi:10.1093/bioinformatics/btag249
Self-citationPinuer, L. P., Zapata, I. T., & Quintanilla, H. B. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-09074-4_42
FlashAttention-T: Towards Fully Tensorized Attention by Exploiting Tensor-Vector Parallelism
Xu, J., Wen, Y., Bi, J., 徐瑞柏, Xu, G., Zhang, R., Li, W., Li, L., Chen, T., Guo, Q., et al. (2026) · ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming · doi:10.1145/3774934.3786425
Černý, M., Májovský, M., Lama, S., Guo, E., Hallak, H., Singh, R., Riva-Cambrin, H. A., Šustek, P., Široká, L., Tullett, I., et al. (2026) · doi:10.2196/preprints.92871
TPLA: Tensor Parallel Latent Attention for Efficient Disaggregated Prefill & Decode Inference
Tang, X., Meng, F., Tang, P., Wang, Y., Yin, D., Sun, X., & Zhang, M. (2026) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3779212.3790237 · Cited by 5
Self-citationWen, J.-L., Li, X.-J., Yao, J., Sun, H.-F., & An, X.-R. (2026) · International Journal of Computational Intelligence Systems · doi:10.1007/s44196-026-01236-9
Self-citationLombardi, R., Destère, A., Dellamonica, J., Gérard, A. O., & Jozwiak, M. (2026) · JMIR Medical Education · doi:10.2196/82702
CARIBOU: Computational AI Research Interface for Bioinformatics, Omics, and Unifying AgentsPreprint
Riffle, D., Shirooni, N., Sureshkumar, P., Vijay, V., & Rose, M. F. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.05.25.727730
Černý, M., Májovský, M., Lama, S., Guo, E., Hallak, H., Singh, R., Riva-Cambrin, H. A., Šustek, P., Široká, L., Tullett, I., et al. (2026) · JMIR Formative Research · doi:10.2196/92871
UniEP: Unified Expert-Parallel MegaKernel MoE for LLM Training
Zheng, S., Zheng, X., Chang, L., & Zhai, J. (2026) · ACM International Symposium on High-Performance Parallel and Distributed Computing (HPDC) · doi:10.1145/3806645.3807818
Self-citationA Road Map for Tamed Interactions with Large Language Models
Scotti, V., Keim, J., Hey, T., Metzger, A., Koziolek, A., & Mirandola, R. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3832776 · Cited by 2
Trinh, F., Paccagnella, A., & Burigana, D. (2026) · Springer eBooks · doi:10.1007/978-3-032-29473-9_24
Lv, N., Zou, S., Chen, J., & Wei, F. (2026) · International Conference on Educational Digitalization and Computer Science in the Guangdong-Hong Kong-Macao Greater Bay Area · doi:10.1145/3828183.3828407
Self-citationLiu, J., Nhu, D., Zhao, X., Jiang, Y., Yan, S., Xue, X., Shen, Y., de Oliveira, G. C., Fong, S., Xu, Q., et al. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-38260-3_50
Self-citationAutoQG: automated evidence-traceable QA via ontology-guided KG
Shen, X., Qi, G., Zhang, P., Yang, D., Yang, Y., Yang, M., Guo, H., & 胡惠康 (2026) · The Knowledge Engineering Review · doi:10.48130/ker-0026-0017
Self-citationQBET: Quantum Based Enhanced Transformer For Efficient Language Modeling
Sharma, V., Shingade, S., & Udmale, S. S. (2026) · Evolving Systems · doi:10.1007/s12530-026-09890-2
Cartridges: Lightweight and general-purpose long context representations via self-study
Eyuboglu, S., Ehrlich, R., Arora, S., Guha, N., Zinsley, D., Liu, E., Tennien, W., Rudra, A., Zou, J., Mirhoseini, A., et al. (2025) · arXiv · doi:10.48550/arxiv.2506.06266
LatentMoE: Toward Optimal Accuracy per FLOP and Parameter in Mixture of Experts
Elango, V., Bhatia, N., Waleffe, R., Shafipour, R., Asida, T., Khattar, A., Assaf, N., Golub, M., Guman, J., Mitra, T., et al. (2026) · arXiv · doi:10.48550/arxiv.2601.18089
LittleBit: Ultra Low-Bit Quantization via Latent Factorization
Lee, B., Kim, D., You, Y., & Kim, Y. (2025) · arXiv · doi:10.48550/arxiv.2506.13771 · Cited by 7
Self-citationA Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.18223 · Cited by 2,090
Self-citationPredictive Multi-Tier Memory Management for KV Cache in Large-Scale GPU Inference
Ganjihal, S. R. (2026) · arXiv · doi:10.48550/arxiv.2604.26968
Metadata
- Authors
- DeepSeek-AI, Liu, A., Feng, B., Wang, B., Wang, B., Liu, B., Zhao, C., Dengr, C., Ruan, C., Dai, D., Guo, D., Yang, D., Chen, D., Ji, D., Li, E., Lin, F., Luo, F., Hao, G., Chen, G., Li, G., Zhang, H., Xu, H., Yang, H., Zhang, H., Ding, H., Xin, H., Gao, H., Li, H., Qu, H., Cai, J. L., Liang, J., Guo, J., Ni, J., Li, J., Chen, J., Yuan, J., Qiu, J., Song, J., Dong, K., Gao, K., Guan, K., Wang, L., Zhang, L., Xu, L., Xia, L., Zhao, L., Zhang, L., Li, M., Wang, M., Zhang, M., Zhang, M., Tang, M., Li, M., Tian, N., Huang, P., Wang, P., Zhang, P., Zhu, Q., Chen, Q., Du, Q., Chen, R. J., Jin, R. L., Ge, R., Pan, R., Xu, R., Chen, R., Li, S. S., Lu, S., Zhou, S., Chen, S., Wu, S., Ye, S., Ma, S., Wang, S., Zhou, S., Yu, S., Zhou, S., Zheng, S., Wang, T., Pei, T., Yuan, T., Sun, T., Xiao, W. L., Zeng, W., An, W., Liu, W., Liang, W., Gao, W., Zhang, W., Li, X. Q., Jin, X., Wang, X., Bi, X., Liu, X., Wang, X., Shen, X., Chen, X., Chen, X., Nie, X., Sun, X., Wang, X., Liu, X., Xie, X., Yu, X., Song, X., Zhou, X., Yang, X., Lu, X., Su, X., Wu, Y., Li, Y. K., Wei, Y. X., Zhu, Y. X., Xu, Y., Huang, Y., Li, Y., Zhao, Y., Sun, Y., Li, Y., Wang, Y., Zheng, Y., Zhang, Y., Xiong, Y., Zhao, Y., He, Y., Tang, Y., Piao, Y., Dong, Y., Tan, Y., Liu, Y., Wang, Y., Guo, Y., Zhu, Y., Wang, Y., Zou, Y., Zha, Y., Ma, Y., Yan, Y., You, Y., Liu, Y., Ren, Z. Z., Ren, Z., Sha, Z., Fu, Z., Huang, Z., Zhang, Z., Xie, Z., Hao, Z., Shao, Z., Wen, Z., Xu, Z., Zhang, Z., Li, Z., Wang, Z., Gu, Z., Li, Z., & Xie, Z.
- Corresponding author
- DeepSeek-AI
- ORCID
- Song, Junxiao
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2405.04434
- OpenAlex
- W4396815229
- Cited by
- 109
- References
- 0
- Citations by year
- 2026: 26 · 2025: 76 · 2024: 7
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- mixture of experts, language models, multi-head latent attention, key-value cache compression, sparse training, inference efficiency, large language models, training cost reduction, open-source models, chatbots
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
