DeepSeek LLM: Scaling Open-Source Language Models with Longtermism
Open accessAbstract
The rapid development of open-source large language models (LLMs) has been truly remarkable. However, the scaling law described in previous literature presents varying conclusions, which casts a dark cloud over scaling LLMs. We delve into the study of scaling laws and present our distinctive findings that facilitate scaling of large scale models in two commonly used open-source configurations, 7B and 67B. Guided by the scaling laws, we introduce DeepSeek LLM, a project dedicated to advancing open-source language models with a long-term perspective. To support the pre-training phase, we have developed a dataset that currently consists of 2 trillion tokens and is continuously expanding. We further conduct supervised fine-tuning (SFT) and Direct Preference Optimization (DPO) on DeepSeek LLM Base models, resulting in the creation of DeepSeek Chat models. Our evaluation results demonstrate that DeepSeek LLM 67B surpasses LLaMA-2 70B on various benchmarks, particularly in the domains of code, mathematics, and reasoning. Furthermore, open-ended evaluations reveal that DeepSeek LLM 67B Chat exhibits superior performance compared to GPT-3.5.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1× · see citing works
“Over the past few years, Large Language Models (LLMs) based on decoder-only Transformers (Vaswani et al., 2017) have increasingly become the cornerstone and pathway to achieving Artificial General Intelligence (AGI).”
BackgroundLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 2× · see citing works
“Results in scaling laws demonstrate that expanding the compute budget continues to yield significant benefits, which further encourages the increase in model scales (Brown et al., 2020; Smith et al., 2022).”
BackgroundMethodLLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602 · cited 16× · see citing works
“Consequently, the LLaMA series has become the de facto benchmark for architecture and performance among open-source models.”
BackgroundMethodInfluentialChain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263 · cited 1× · see citing works
“As a significant proportion of the SFT instances are in the CoT format Wei et al. (2022), the chat models demonstrate slight improvements in reasoning tasks, such as BBH and NaturalQuestions.”
BackgroundPIQA: Reasoning about Physical Commonsense in Natural Language
Bisk, Y., Zellers, R., Bras, R. L., Gao, J.-F., & Choi, Y. (2019) · AAAI Conference on Artificial Intelligence · doi:10.48550/arxiv.1911.11641 · Cited by 3,856
Transformer-XL: Attentive language models beyond a fixed-length context
Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q., & Salakhutdinov, R. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1285 · Cited by 3,169 · cited 12× · see citing works
“By predicting the next word in continuous text, LLMs undergo self-supervised pre-training on massive datasets, enabling them to achieve various purposes and possess many abilities, such as novel creation, text summarization, code completion, and more.”
BackgroundInfluentialTraining Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117 · cited 14× · see citing works
“The value of SFT lies in the ability to learn to achieve comparable scores to the base model’s few-shot setting in the chat model’s zero-shot setting, which is aligned with real scenarios.”
MethodInfluentialCan a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering
Mihaylov, T., Clark, P., Khot, T., & Sabharwal, A. (2018) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1260 · Cited by 2,998 · cited 4× · see citing works
“- flowers blooming - grass turning brown - trees growing - blossoms blooming Table 36 | An example of OpenBookQA.”
InfluentialLlama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604 · see citing works
Self-citationAccurate, Large Minibatch SGD: Training ImageNet in 1 Hour
Goyal, P., Dollár, P., Girshick, R., Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y., & He, K. (2017) · arXiv · doi:10.48550/arxiv.1706.02677 · Cited by 2,294 · cited 1× · see citing works
“Early works (Goyal et al., 2017; McCandlish et al., 2018; Shallue et al., 2019; Smith et al., 2017; Zhang et al., 2019) provided some empirical observations for setting batch size and learning rate, but we found these observations have limited applicability in our preliminary experiments.”
ResultNatural Questions: A benchmark for question answering research
Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., et al. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00276 · Cited by 2,069 · cited 1× · see citing works
“Closed-book question answering datasets including TriviaQA (Joshi et al., 2017) and Natu-ralQuestions (Kwiatkowski et al., 2019).”
MethodGLM: General Language Model Pretraining with Autoregressive Blank Infilling
Du, Z.-X., Qian, Y., Liu, X., Ding, M., Qiu, J.-Z., Yang, Z.-L., & Tang, J. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2022.acl-long.26 · Cited by 2,032 · cited 4× · see citing works
“For instance, ChatGLM3 achieves a score of 52.4 on MBPP, a code testset, which is close to DeepSeek 67B.”
BackgroundMethodInfluentialSelf-citationRoFormer: Enhanced Transformer with Rotary Position Embedding
Su, J., Lu, Y., Pan, S., Murtadha, A., Wen, B., & Liu, Y. (2021) · arXiv · doi:10.48550/arxiv.2104.09864 · Cited by 1,952 · cited 1× · see citing works
“It also incorporates Rotary Embedding (Su et al., 2024) for positional encoding.”
MethodSelf-citationJudging LLM-as-a-Judge with MT-Bench and Chatbot Arena
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.05685 · Cited by 1,888 · see citing works
Self-citationEfficient Memory Management for Large Language Model Serving with PagedAttention
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., & Stoica, I. (2023) · Proceedings of the 29th Symposium on Operating Systems Principles · doi:10.1145/3600006.3613165 · Cited by 1,817 · cited 5× · see citing works
“These datasets include HellaSwag, PIQA, WinoGrande, RACE-Middle, RACE-High, MMLU, ARC-Easy, ARC-Challenge, OpenBookQA, CHID, C-Eval, CMMLU, C3 and CCPM.”
MethodInfluentialSelf-citationScaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · Cited by 1,749 · cited 7× · see citing works
“However, as shown in Table 4, early works (Hoffmann et al., 2022; Kaplan et al., 2020) on the optimal model/data scaling-up allocation strategy have shown varying conclusions, raising doubts about the general applicability of scaling laws.”
BackgroundMethodResultInfluentialTriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Joshi, M., Choi, E., Weld, D., & Zettlemoyer, L. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1147 · Cited by 1,690 · cited 4× · see citing works
“Closed-book question answering datasets including TriviaQA (Joshi et al., 2017) and Natu-ralQuestions (Kwiatkowski et al., 2019).”
MethodInfluentialEvaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436 · cited 6× · see citing works
“For instance, HumanEval and GSM8K scores are improved by over 20 points.”
BackgroundInfluentialFlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
Dao, T., Fu, D. Y., Ermon, S., Rudra, A., & Ré, C. (2022) · arXiv · doi:10.48550/arxiv.2205.14135 · Cited by 1,214 · cited 1× · see citing works
“We also leverage the flash attention (Dao, 2023; Dao et al., 2022) technique to improve hardware utilization.”
MethodDon't Decay the Learning Rate, Increase the Batch SizePreprint
Smith, S. L., Kindermans, P.-J., Ying, C., & Le, Q. V. (2017) · International Conference on Learning Representations · doi:10.48550/arxiv.1711.00489 · Cited by 1,180 · cited 1× · see citing works
“Early works (Goyal et al., 2017; McCandlish et al., 2018; Shallue et al., 2019; Smith et al., 2017; Zhang et al., 2019) provided some empirical observations for setting batch size and learning rate, but we found these observations have limited applicability in our preliminary experiments.”
ResultRACE: Large-scale ReAding Comprehension Dataset From Examinations
Lai, G., Xie, Q., Liu, H., Yang, Y., & Hovy, E. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1082 · Cited by 1,044 · cited 2× · see citing works
“These datasets include HellaSwag, PIQA, WinoGrande, RACE-Middle, RACE-High, MMLU, ARC-Easy, ARC-Challenge, OpenBookQA, CHID, C-Eval, CMMLU, C3 and CCPM.”
Training Compute-Optimal Large Language Models
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. D. L., Hendricks, L. A., Welbl, J., Clark, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2176 · Cited by 932 · cited 7× · see citing works
“However, as shown in Table 4, early works (Hoffmann et al., 2022; Kaplan et al., 2020) on the optimal model/data scaling-up allocation strategy have shown varying conclusions, raising doubts about the general applicability of scaling laws.”
BackgroundMethodInfluentialTraining Compute-Optimal Large Language Models
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. D. L., Hendricks, L. A., Welbl, J., Clark, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.15556 · Cited by 929 · cited 7× · see citing works
“However, as shown in Table 4, early works (Hoffmann et al., 2022; Kaplan et al., 2020) on the optimal model/data scaling-up allocation strategy have shown varying conclusions, raising doubts about the general applicability of scaling laws.”
BackgroundMethodInfluentialMegatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., & Catanzaro, B. (2019) · arXiv · doi:10.48550/arxiv.1909.08053 · Cited by 893 · cited 1× · see citing works
“Data parallelism, tensor parallelism, sequence parallelism, and 1F1B pipeline parallelism are integrated into this framework as done in Mega-tron (Korthikanti et al., 2023; Narayanan et al., 2021; Shoeybi et al., 2019).”
MethodMetaMath: Bootstrap Your Own Mathematical Questions for Large Language ModelsPreprint
Yu, L., Jiang, W., Shi, H., Yu, J., Liu, Z., Zhang, Y., Kwok, J. T.-Y., Li, Z., Weller, A., & Liu, W. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2309.12284 · Cited by 844 · see citing works
Self-citationHellaSwag: Can a Machine Really Finish Your Sentence?
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., & Choi, Y. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1472 · Cited by 830 · cited 3× · see citing works
“These particular tasks typically involve cloze tasks or sentence completion tasks, such as HellaSwag.”
Direct Preference Optimization: Your Language Model is Secretly a Reward Model
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., & Finn, C. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-2338 · Cited by 686 · cited 12× · see citing works
“To tackle the problem, we tried two-stage fine-tuning and DPO (Rafailov et al., 2023), both of which could almost keep the benchmark score and reduce the repetition significantly.”
MethodInfluentialEfficient large-scale language model training on GPU clusters using megatron-LM
Narayanan, D., Shoeybi, M., Casper, J., LeGresley, P., Patwary, M., Korthikanti, V., Vainbrand, D., Kashinkunti, P., Bernauer, J., Catanzaro, B., et al. (2021) · Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis · doi:10.1145/3458817.3476209 · Cited by 639 · cited 1× · see citing works
“Data parallelism, tensor parallelism, sequence parallelism, and 1F1B pipeline parallelism are integrated into this framework as done in Mega-tron (Korthikanti et al., 2023; Narayanan et al., 2021; Shoeybi et al., 2019).”
MethodZeRO: Memory Optimization Towards Training A Trillion Parameter ModelsPreprint
Rajbhandari, S., Rasley, J., Ruwase, O., & He, Y. (2019) · arXiv.org · doi:10.48550/arxiv.1910.02054 · Cited by 612 · cited 1× · see citing works
“ZeRO-1 (Rajbhandari et al., 2020) is exploited to partition optimizer states over data parallel ranks.”
BackgroundSelf-citationTraining Verifiers to Solve Math Word Problems
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. (2021) · arXiv · doi:10.48550/arxiv.2110.14168 · Cited by 575 · cited 3× · see citing works
“Math datasets including GSM8K (Cobbe et al., 2021), MATH (Hendrycks et al., 2021) and CMath (Wei et al., 2023).”
BackgroundMethodReducing Activation Recomputation in Large Transformer ModelsPreprint
Korthikanti, V., Casper, J., Lym, S., McAfee, L. C., Andersch, M., Shoeybi, M., & Catanzaro, B. (2022) · Conference on Machine Learning and Systems · doi:10.48550/arxiv.2205.05198 · Cited by 571 · cited 1× · see citing works
“Data parallelism, tensor parallelism, sequence parallelism, and 1F1B pipeline parallelism are integrated into this framework as done in Mega-tron (Korthikanti et al., 2023; Narayanan et al., 2021; Shoeybi et al., 2019).”
MethodThe Pile: An 800GB Dataset of Diverse Text for Language Modeling
Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al. (2021) · arXiv · doi:10.48550/arxiv.2101.00027 · Cited by 492 · cited 3× · see citing works
“Language modeling datasets including Pile (Gao et al., 2020).”
BackgroundCLUE: A Chinese Language Understanding Evaluation BenchmarkPreprint
Xu, L., Zhang, X., Li, L., Hu, H., Cao, C., Liu, W., Li, J., Li, Y., Sun, K., Xu, Y., et al. (2020) · International Conference on Computational Linguistics · doi:10.48550/arxiv.2004.05986 · Cited by 485 · cited 2× · see citing works
“We apply generation-based evaluation for TriviaQA, NaturalQuestions, DROP, MATH, GSM8K, HumanEval, MBPP, BBH, AGIEval, CLUEWSC, and CMath.”
Self-citationAn Empirical Model of Large-Batch TrainingPreprint
McCandlish, S., Kaplan, J., Amodei, D., & OpenAI Dota Team (2018) · arXiv.org · doi:10.48550/arxiv.1812.06162 · Cited by 466 · cited 2× · see citing works
“Early works (Goyal et al., 2017; McCandlish et al., 2018; Shallue et al., 2019; Smith et al., 2017; Zhang et al., 2019) provided some empirical observations for setting batch size and learning rate, but we found these observations have limited applicability in our preliminary experiments.”
ResultThink you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., & Tafjord, O. (2018) · arXiv · doi:10.48550/arxiv.1803.05457 · Cited by 462 · cited 3× · see citing works
“These datasets include HellaSwag, PIQA, WinoGrande, RACE-Middle, RACE-High, MMLU, ARC-Easy, ARC-Challenge, OpenBookQA, CHID, C-Eval, CMMLU, C3 and CCPM.”
GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebrón, F., & Sanghai, S. (2023) · arXiv · doi:10.48550/arxiv.2305.13245 · Cited by 447 · cited 1× · see citing works
“To optimize inference cost, the 67B model uses Grouped-Query Attention (GQA) (Ainslie et al., 2023) instead of the traditional Multi-Head Attention (MHA).”
MethodDeep Learning Scaling is Predictable, Empirically
Hestness, J., Narang, S., Ardalani, N., Diamos, G., Jun, H., Kianinejad, H., Patwary, M. M. A., Yang, Y., & Zhou, Y. (2017) · arXiv · doi:10.48550/arxiv.1712.00409 · Cited by 427 · cited 1× · see citing works
“Research on scaling laws (Hestness et al., 2017) predates the emergence of large language models.”
BackgroundDROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs
Dua, D., Wang, Y., Dasigi, P., Stanovsky, G., Singh, S., & Gardner, M. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1246 · Cited by 393 · cited 2× · see citing works
“We apply generation-based evaluation for TriviaQA, NaturalQuestions, DROP, MATH, GSM8K, HumanEval, MBPP, BBH, AGIEval, CLUEWSC, and CMath.”
Self-citationProgram Synthesis with Large Language Models
Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07732 · Cited by 388 · cited 4× · see citing works
“For instance, ChatGLM3 achieves a score of 52.4 on MBPP, a code testset, which is close to DeepSeek 67B.”
InfluentialJiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. D. L., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.06825 · Cited by 340 · cited 1× · see citing works
“These products have significantly raised the community’s expectations for the capabilities of open-source LLMs, consequently inspiring a series of work (Bai et al., 2023; Du et al., 2022; Jiang et al., 2023; Touvron et al., 2023a,b; Yang et al., 2023).”
BackgroundBai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.16609 · Cited by 332 · cited 14× · see citing works
“We can find that our model has a higher safety score than both ChatGPT and GPT-4, placing it amongst the ranks of the safest models.”
BackgroundMethodInfluentialSelf-citationCLUE: A Chinese Language Understanding Evaluation Benchmark
Xu, L. J., Hu, H., Zhang, X., Lü, L., Cao, C., Li, Y., Xu, Y., Sun, K., Yu, D., Yu, C., et al. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.419 · Cited by 307 · see citing works
Self-citationMeasuring Mathematical Problem Solving With the MATH Dataset
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., & Steinhardt, J. (2021) · arXiv · doi:10.48550/arxiv.2103.03874 · Cited by 306 · cited 4× · see citing works
“Math datasets including GSM8K (Cobbe et al., 2021), MATH (Hendrycks et al., 2021) and CMath (Wei et al., 2023).”
BackgroundMethodInfluentialGLU Variants Improve Transformer
Shazeer, N. (2020) · arXiv · doi:10.48550/arxiv.2002.05202 · Cited by 303 · cited 2× · see citing works
“Params The micro design of DeepSeek LLM largely follows the design of LLaMA (Touvron et al., 2023a,b), adopting a Pre-Norm structure with RMSNorm (Zhang and Sennrich, 2019) function and using SwiGLU (Shazeer, 2020) as the activation function for the Feed-Forward Network (FFN), with an intermediate layer dimension of 𝑚…”
MethodMeasuring Massive Multitask Language Understanding
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., & Steinhardt, J. (2020) · arXiv · doi:10.48550/arxiv.2009.03300 · Cited by 291 · cited 6× · see citing works
“These datasets include HellaSwag, PIQA, WinoGrande, RACE-Middle, RACE-High, MMLU, ARC-Easy, ARC-Challenge, OpenBookQA, CHID, C-Eval, CMMLU, C3 and CCPM.”
MethodInfluentialSmith, S., Patwary, M., Norick, B., LeGresley, P., Rajbhandari, S., Casper, J., Liu, Z., Prabhumoye, S., Zerveas, G., Korthikanti, V., et al. (2022) · arXiv · doi:10.48550/arxiv.2201.11990 · Cited by 286 · cited 1× · see citing works
“Results in scaling laws demonstrate that expanding the compute budget continues to yield significant benefits, which further encourages the increase in model scales (Brown et al., 2020; Smith et al., 2022).”
BackgroundSelf-citationDo-Not-Answer: A Dataset for Evaluating Safeguards in LLMsPreprint
Wang, Y., Li, H., Han, X., Nakov, P., & Baldwin, T. (2023) · arXiv.org · doi:10.48550/arxiv.2308.13387 · Cited by 236 · see citing works
Self-citationMeasuring the Effects of Data Parallelism on Neural Network Training
Shallue, C. J., Lee, J., Antognini, J., Sohl-Dickstein, J., Frostig, R., & Dahl, G. E. (2018) · arXiv · doi:10.48550/arxiv.1811.03600 · Cited by 229 · cited 1× · see citing works
“Early works (Goyal et al., 2017; McCandlish et al., 2018; Shallue et al., 2019; Smith et al., 2017; Zhang et al., 2019) provided some empirical observations for setting batch size and learning rate, but we found these observations have limited applicability in our preliminary experiments.”
ResultAGIEval: A Human-Centric Benchmark for Evaluating Foundation Models
Zhong, W., Cui, R., Guo, Y., Liang, Y., Lu, S., Wang, Y., Saied, A., Chen, W., & Duan, N. (2023) · arXiv · doi:10.48550/arxiv.2304.06364 · Cited by 211 · cited 2× · see citing works
“We apply generation-based evaluation for TriviaQA, NaturalQuestions, DROP, MATH, GSM8K, HumanEval, MBPP, BBH, AGIEval, CLUEWSC, and CMath.”
Self-citationRoot Mean Square Layer Normalization
Zhang, B., & Sennrich, R. (2019) · Curran Associates · doi:10.5167/uzh-177483 · Cited by 203 · cited 3×
“Params The micro design of DeepSeek LLM largely follows the design of LLaMA (Touvron et al., 2023a,b), adopting a Pre-Norm structure with RMSNorm (Zhang and Sennrich, 2019) function and using SwiGLU (Shazeer, 2020) as the activation function for the Feed-Forward Network (FFN), with an intermediate layer dimension of 𝑚…”
MethodResultSelf-citationWhich Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic ModelPreprint
Zhang, G., Li, L.-L., Nado, Z., Martens, J., Sachdeva, S., Dahl, G., Shallue, C. J., & Grosse, R. (2019) · Neural Information Processing Systems · doi:10.48550/arxiv.1907.04164 · Cited by 198 · cited 1× · see citing works
“Early works (Goyal et al., 2017; McCandlish et al., 2018; Shallue et al., 2019; Smith et al., 2017; Zhang et al., 2019) provided some empirical observations for setting batch size and learning rate, but we found these observations have limited applicability in our preliminary experiments.”
ResultPenedo, G., Malartic, Q., Hesslow, D., Cojocaru, R., Cappelli, A., Alobeidli, H., Pannier, B., Almazrouei, E., & Launay, J. (2023) · arXiv · doi:10.48550/arxiv.2306.01116 · Cited by 164 · cited 1× · see citing works
“We have gained valuable insights from reputable sources such as (Computer, 2023; Gao et al., 2020; Penedo et al., 2023; Touvron et al., 2023a).”
BackgroundScaling Laws for Autoregressive Generative Modeling
Henighan, T., Kaplan, J., Katz, M., Chen, M., Hesse, C., Jackson, J., Jun, H., Brown, T. B., Dhariwal, P., Gray, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.14701 · Cited by 150 · cited 7× · see citing works
“However, as shown in Table 4, early works (Hoffmann et al., 2022; Kaplan et al., 2020) on the optimal model/data scaling-up allocation strategy have shown varying conclusions, raising doubts about the general applicability of scaling laws.”
BackgroundMethodResultInfluentialFlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
Dao, T. (2023) · arXiv · doi:10.48550/arxiv.2307.08691 · Cited by 143 · cited 1× · see citing works
“We also leverage the flash attention (Dao, 2023; Dao et al., 2022) technique to improve hardware utilization.”
MethodInvestigating Prior Knowledge for Challenging Chinese Machine Reading Comprehension
Sun, K., Yu, D., Yu, D., & Cardie, C. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00305 · Cited by 142 · cited 1× · see citing works
“RACE Lai et al. (2017) and DROP (Dua et al., 2019), C3 (Sun et al., 2019).”
ChID: A Large-scale Chinese IDiom Dataset for Cloze Test
Zheng, C., Huang, M., & Sun, A. (2019) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1075 · Cited by 114 · cited 3× · see citing works
“However, tasks like CHID, which involve evaluating the usage of Chinese idioms, require the model to consume a significant number of Chinese tokens during pre-training.”
BackgroundAlignBench: Benchmarking Chinese Alignment of Large Language ModelsPreprint
Liu, X., Lei, X., Wang, S., Huang, Y., Feng, A., Wen, B.-S., Cheng, J., Ke, P., Xu, Y.-F., Tam, W., et al. (2023) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2311.18743 · Cited by 85 · cited 5× · see citing works
“For Chinese open-ended evaluation, we tested the comprehensive of our chat model in different domains on a high-quality open-ended question testset AlignBench (Liu et al., 2023).”
MethodInfluentialSelf-citationChallenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them
Suzgun, M., Scales, N., Schärli, N., Gehrmann, S., Tay, Y., Chung, H. W., Chowdhery, A., Le, Q. V., Chi, E. H., Zhou, D., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.09261 · Cited by 44 · cited 5× · see citing works
“As a significant proportion of the SFT instances are in the CoT format Wei et al. (2022), the chat models demonstrate slight improvements in reasoning tasks, such as BBH and NaturalQuestions.”
InfluentialInstruction-Following Evaluation for Large Language Models
Zhou, J., Lu, T., Mishra, S., Brahma, S., Basu, S., Luan, Y., Zhou, D., & Hou, L. (2023) · arXiv · doi:10.48550/arxiv.2311.07911 · Cited by 39 · cited 1× · see citing works
“Instruction Following Evaluation: On Nov 15th, 2023, Google released an instruction following the evaluation dataset (Zhou et al., 2023).”
MethodPIQA: Reasoning about Physical Commonsense in Natural Language
Bisk, Y., Zellers, R., Le bras, R., Gao, J., & Choi, Y. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6239 · Cited by 37 · see citing works
WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct
Luo, H., Sun, Q., Xu, C., Zhao, P., Lou, J., Tao, C., Geng, X., Lin, Q., Chen, S., Tang, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.09583 · Cited by 33 · see citing works
Self-citationToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving
Gou, Z., Shao, Z., Gong, Y., Shen, Y., Yang, Y., Huang, M., Duan, N., & Chen, W. (2023) · arXiv · doi:10.48550/arxiv.2309.17452 · Cited by 24 · cited 3× · see citing works
“We have conducted a comparative analysis of our model against various baseline models of different sizes, namely Qwen 72B Chat (Bai et al., 2023), ChatGLM3 (Du et al., 2022), Baichuan2 (Yang et al., 2023), and Yi-34B Chat.”
BackgroundMethodSelf-citationCamels in a Changing Climate: Enhancing LM Adaptation with Tulu 2
Ivison, H., Wang, Y., Pyatkin, V., Lambert, N., Peters, M., Dasigi, P., Jang, J., Wadden, D., Smith, N. A., Beltagy, I., et al. (2023) · arXiv · doi:10.48550/arxiv.2311.10702 · Cited by 8 · cited 1× · see citing works
“As illustrated in (Ivison et al., 2023), and achieves 8.35 score comparable with GPT-3.”
ResultInfluentialSelf-citationC-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models
Huang, Y., Bai, Y., Zhu, Z., Zhang, J., Zhang, J., Su, T., Liu, J., Lv, C., Zhang, Y., Lei, J., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.08322 · cited 5×
“These datasets include HellaSwag, PIQA, WinoGrande, RACE-Middle, RACE-High, MMLU, ARC-Easy, ARC-Challenge, OpenBookQA, CHID, C-Eval, CMMLU, C3 and CCPM.”
MethodInfluentialSelf-citationCMMLU: Measuring massive multitask language understanding in Chinese
Li, H., Zhang, Y., Koto, F., Yang, Y., Zhao, H., Gong, Y., Duan, N., & Baldwin, T. (2023) · arXiv · doi:10.48550/arxiv.2306.09212
Self-citation
Flagged first, then by citations
6 of the 71 references couldn’t be matched to a known paper.
Research works printed but not linked: 5
- F. i, M. Suzgun, M. Freitag, X. Wang, S. Srivats, S. Vosoughi, H. W. Chung, Y. Tay, S. Ruder, D. Zhou, D. Das, and J. Wei. Language models are multilingual chain-of-thought reasoners. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https://openreview.net/pdf?id= fR3wGCk-IXp.
- I. Loshchilov and F. Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. · arXiv
- OpenAI. GPT4 technical report. arXiv preprint arXiv:2303.08774, 2023. · arXiv
- A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
- S. Rajbhandari, J. Rasley, O. Ruwase, and Y. He. Zero: Memory optimizations toward training tril- lion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1-16. IEEE, 2020.
Cited by
Sorted by citations
How far are we to GPT-4V? Closing the gap to commercial multimodal models with open-source suites
Chen, Z., Wang, W., Tian, H., Ye, S., Gao, Z., Cui, E., Tong, W., Hu, K., Luo, J., Ma, Z., et al. (2024) · Science China Information Sciences · doi:10.1007/s11432-024-4231-5 · Cited by 437 · see citing works
“Large language models (LLMs) [3, 6, 9, 10, 13, 26, 99, 101, 103, 107, 108, 117, 118, 136] have greatly advanced AGI by enabling complex language tasks previously thought human-exclusive.”
Self-citationDeepSeek-AI, Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., et al. (2024) · arXiv · doi:10.48550/arxiv.2412.19437 · Cited by 294 · see citing works
Self-citationA Survey on Mixture of Experts in Large Language Models
Cai, W., Jiang, J., Wang, F., Tang, J., Kim, S., & Huang, J. (2025) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2025.3554028 · Cited by 334 · see citing works
“Specifically, DeepSeekMoE-145B employs a reduced intermediate hidden dimension at one-eighth that of its dense FFN counterpart, increasing both the number of experts (from 16 to 128) and the number of active experts (from top-2 to top-16) by a factor of eight.”
InfluentialExploring DeepSeek: A Survey on Advances, Applications, Challenges and Future Directions
Deng, Z., Ma, W., Han, Q., Zhou, W., Zhu, X., Wen, S., & Xiang, Y. (2025) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2025.125498 · Cited by 173 · see citing works
Temsah, A., Alhasan, K., Altamimi, I., Jamal, A., Al-Eyadhy, A., Malki, K. H., & Temsah, M.-H. (2025) · Cureus · doi:10.7759/cureus.79221 · Cited by 112 · see citing works
“The open-source nature of DeepSeek's DeepThink R1 may empower innovation among international scholars [9].”
DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
DeepSeek-AI, Liu, A., Feng, B., Wang, B., Wang, B., Liu, B., Zhao, C., Dengr, C., Ruan, C., Dai, D., et al. (2024) · arXiv · doi:10.48550/arxiv.2405.04434 · Cited by 109 · see citing works
Self-citationDeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y. K., Wu, Y., et al. (2024) · arXiv · doi:10.48550/arxiv.2402.03300 · Cited by 101 · see citing works
Self-citationUAVs meet LLMs: Overviews and perspectives towards agentic low-altitude mobility
Tian, Y., Lin, F., Li, Y., Zhang, T., Zhang, Q., Fu, X., Huang, J., Dai, X., Wang, Y., Tian, C., et al. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103158 · Cited by 102 · see citing works
Self-citationA comprehensive survey on integrating large language models with knowledge-based methods
Yang, W., Some, L., Bain, M., & Kang, B. H. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113503 · Cited by 88 · see citing works
“Second, computational efficiency and accessibility have been key drivers in the development of open-source LLMs. DeepSeek LLM [16] exemplifies this trend, offering a scalable and transparent alternative to proprietary models.”
Accelerating battery innovation: AI-powered molecular discovery
Gao, Y., Chen, X., Yuan, Y., Chen, Y., Niu, Y.-L., Yao, N., Gao, Y.-B., Li, W. Z., & Zhang, Q. (2025) · Chemical Society Reviews · doi:10.1039/d5cs00053j · Cited by 61 · see citing works
Rahman, A., Mahir, S. H., Tashrif, M. T. A., Karim, M. A., Aishi, A. A., Kundu, D., Debnath, T., Moududi, M. A. A., Eidmum, M. Z. A., Miah, A. S. M., et al. (2025) · Systems and Soft Computing · doi:10.1016/j.sasc.2025.200396 · Cited by 81 · see citing works
Evaluation of LLM-based chatbots for OSINT-based Cyber Threat Awareness
Shafee, S., Bessani, A., & Ferreira, P. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.125509 · Cited by 83 · see citing works
“One key contributing factor to this accuracy gap is the significantly larger context window size employed by the ChatGPT.”
DeepSeek vs. ChatGPT: prospects and challenges
Jin, I., Tangsrivimol, J. A., Darzidehkalani, E., Virk, H. U. H., Wang, Z., Egger, J., Hacking, S. M., Glicksberg, B. S., Strauß, M., & Krittanawong, C. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2025.1576992 · Cited by 60 · see citing works
How can artificial intelligence transform the training of medical students and physicians?
Ning, Y., Ong, J. C. L., Cheng, H., Wang, H., Ting, D. S. W., Tham, Y. C., Wong, T. Y., & Liu, N. (2025) · The Lancet Digital Health · doi:10.1016/j.landig.2025.100900 · Cited by 57 · see citing works
DeepSeek: Paradigm Shifts and Technical Evolution in Large AI Models
Xiong, L., Wang, H., Chen, X., Sheng, L., Xiong, Y., Liu, J., Xiao, Y., Chen, H., Han, Q., & Tang, Y. S. (2025) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2025.125495 · Cited by 46 · see citing works
“Lastly, DeepSeek-LLM (V1) [39] Nov 2023 7B, 67B 4K 2T DeepSeek-MoE [40] Jan 2024 16B 4K 2T DeepSeekMath [41] Apr 2024 1.3B 4K 150B DeepSeek-V2 [42] May 2024 236B, 21B actived per token 128K 8.1T DeepSeek-V3 [38] Dec 2024 671B, 37B actived per token 128K 14.8T DeepSeek-R1 [1] Jan 2025 671B 128K -…”
Self-citationInsights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architectures
Zhao, C., Deng, C., Ruan, C., Dai, D., Gao, H., Li, J., Zhang, L., Huang, P., Zhou, S., Ma, S., et al. (2025) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1145/3695053.3731412 · Cited by 60 · see citing works
Self-citationGeneral Place Recognition Survey: Toward Real-World Autonomy
Yin, P., Jiao, J., Zhao, S., Xu, L., Huang, G. P., Choset, H., Scherer, S., & Han, J. (2025) · IEEE Transactions on Robotics · doi:10.1109/tro.2025.3550771 · Cited by 41 · see citing works
“This systematic review is particularly pertinent given transformative developments in three key areas: 1) foundational language models [9], [10] demonstrating unprecedented reasoning capabilities; 2) neural scene representation frameworks [11] enabling photorealistic environment reconstruction; and…”
RTLRewriter: Methodologies for Large Models aided RTL Code Optimization
Yao, X., Wang, Y., Li, X., Lian, Y., Chen, R., Chen, L., Yuan, M., Xu, H. L., & Yu, B. (2024) · doi:10.1145/3676536.3676775 · Cited by 26 · see citing works
“Additionally, we incorporate the traditional TF-IDF [44] representation for code search to enhance retrieval effectiveness.”
Self-citationTraining Compute-Optimal Protein Language ModelsPreprint
Cheng, X., Chen, B., Li, P., Gong, J., Tang, J., & Song, L. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.06.06.597716 · Cited by 21 · see citing works
“We employed the IsoFLOPs profiling approach [36, 9] the minimal variations in the final loss among a set of ( N, D ) configurations, we classify these configurations as operating under "IsoLoss" conditions (see Appendix K Figure A15), considered optimal for training.”
Fire-Flyer AI-HPC: A Cost-Effective Software-Hardware Co-Design for Deep Learning
An, W., Bi, X., Chen, G.-T., Chen, S., Deng, C., Ding, H., Dong, K., Du, Q., Gao, W., Guan, K., et al. (2024) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1109/sc41406.2024.00089 · Cited by 26 · see citing works
“Consequently, Fire-Flyer 2 AI-HPC now supports the training tasks of the DeepSeek-AI series large language models [72], [97]–[101].”
Self-citationXie, C., Gao, H., Huang, Y., Xue, Z., Xu, C. B., & Dai, K. (2025) · Earthquake research advances · doi:10.1016/j.eqrea.2025.100378 · Cited by 21 · see citing works
Self-citationFoundation models for radiology—the position of the AI for Health Imaging (AI4HI) network
de Almeida, J. G., Alberich, L. C., Tsakou, G., Kostas Marias, A., Tsiknakis, M., Lekadir, K., Martí‐Bonmatí, L., & Papanikolaou, N. K. (2025) · Insights into Imaging · doi:10.1186/s13244-025-02056-9 · Cited by 21 · see citing works
“The GPT family of models (starting with GPT-2 [122]), LLaMa [118], DeepSeek [124], Gemma 3 [125] Vision language models (VLMs) Vision language models combine the text-generation capabilities of LLMs with generic visual understanding.”
Toward self-driving laboratory 2.0 for chemistry and materials discovery
Lee, H., Yoo, H. J., Jang, H. S., Park, B., Park, Y. J., & Han, S. S. (2026) · Materials Horizons · doi:10.1039/d5mh01984b · Cited by 20 · see citing works
Towards Medical Complex Reasoning with LLMs through Medical Verifiable Problems
Chen, J., Cai, Z., Ji, K., Wang, X., Liu, W., Wang, R., & Wang, B. (2025) · Findings of the Association for Computational Linguistics: ACL 2025 · doi:10.18653/v1/2025.findings-acl.751 · Cited by 14 · see citing works
Self-citationAbstractive text summarization: A comprehensive survey of techniques, systems, and challenges
Almohaimeed, N., & Azmi, A. M. (2025) · Computer Science Review · doi:10.1016/j.cosrev.2025.100762 · Cited by 17 · see citing works
Large Language Models for Combinatorial Optimization: A Systematic Review
Ros, F. D., Soprano, M., Gaspero, L. D., & Roitero, K. (2026) · ACM Computing Surveys · doi:10.1145/3801961 · Cited by 24 · see citing works
Nasution, A., Onan, A., Murakami, Y., Monika, W., & Hanafiah, A. (2025) · IEEE Access · doi:10.1109/access.2025.3574629 · Cited by 17 · see citing works
“DeepSeek Series : DeepSeek R1 (1.5B, 7B, 8B, 14B, 32B, 70B) - a family of open models aimed at scaling laws and long-term open-source model development [32].”
OLMo: Accelerating the Science of Language Models
Groeneveld, D., Beltagy, I., Walsh, P., Bhagia, A., Kinney, R., Tafjord, O., Jha, A. H., Ivison, H., Magnusson, I., Wang, Y., et al. (2024) · arXiv · doi:10.48550/arxiv.2402.00838 · Cited by 10 · see citing works
Self-citationHotPrefix: Hotness-Aware KV Cache Scheduling for Efficient Prefix Sharing in LLM Inference Systems
Li, Y., Gu, R., Huan, C., Wang, Z., Yao, R., Tian, C., & Chen, G. (2025) · Proceedings of the ACM on Management of Data · doi:10.1145/3749168 · Cited by 9 · see citing works
Self-citationSun, Y., & Zhou, T. (2025) · IEEE Access · doi:10.1109/access.2025.3591447 · Cited by 12 · see citing works
Self-citationDesigning Incremental Knowledge Enrichment in Generative Pre-trained TransformersPreprint
Kowalczyk, E. A., Nowakowski, M., & Brzezińska, Z. (2024) · Research Square · doi:10.21203/rs.3.rs-4191646/v1 · Cited by 9 · see citing works
Huang, Z., Kaur Sekhon, V., Sadeghian, R., Vaida, M. L., Jo, C., McCormick, B. A., Ward, D. V., Bucci, V., & Haran, J. P. (2025) · IEEE Access · doi:10.1109/access.2025.3599857 · Cited by 10 · see citing works
“Other transformer-based LLM families, such as LLaMA, Gemini, Claude, and DeepSeek, have also demonstrated powerful structured and unstructured biomedical data processing [4], [5], [6], [7], [8], [9], [10].”
Zhang, C., You, Y., Wang, N., Park, J., & Zhang, L. (2025) · IEEE Journal on Emerging and Selected Topics in Circuits and Systems · doi:10.1109/jetcas.2025.3575272 · Cited by 11 · see citing works
Self-citationTop Pass: improve code generation by pass@k-maximized code ranking
Lyu, Z., Li, X., Xie, Z., & Li, M. (2025) · Frontiers of Computer Science · doi:10.1007/s11704-024-40415-9 · Cited by 9 · see citing works
Self-citationIlse, B., & Blackwood, F. (2024) · Research Square · doi:10.21203/rs.3.rs-4895456/v1 · Cited by 8 · see citing works
Large language models for porous materials: from text mining to autonomous laboratory
Han, S., Bae, T., Kim, J., Kim, Y.-H., & Kim, J. (2026) · Digital Discovery · doi:10.1039/d5dd00578g · Cited by 7 · see citing works
BRIDGE: benchmarking large language models for understanding real-world clinical practice texts
Wu, J., Gu, B., Zhou, R., Xie, K., Snyder, D., Jiang, Y., Carducci, V., Wyss, R., Desai, R. J., Alsentzer, E., et al. (2026) · Nature Biomedical Engineering · doi:10.1038/s41551-026-01719-2 · Cited by 14 · see citing works
DrLLM: Prompt-Enhanced Distributed Denial-of-Service Resistance Method with Large Language Models
Yin, Z., Liu, S., & Xu, G. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889735 · Cited by 6 · see citing works
“We evaluate DrLLM in zero-shot scenarios by comparing it with four state-of-the-art LLMs, including GPT-4o-mini [29], Llama3-70b [4], Deepseek-chat-v2 [5] and Qwen2-57b-a14b-instruct [30].”
BackgroundMethodMaslinski, J., Grasfield, R. H., Awasthi, R., Mishra, S. V., Mahapatra, D., & Mathur, P. (2025) · Cureus · doi:10.7759/cureus.82397 · Cited by 5 · see citing works
“These laws suggest that the LLMs’ performance improves with increases in these three factors: model size, data amount, and computational power [26].”
Cheng, Y., & Zhang, H. J. (2025) · ACS ES&T Water · doi:10.1021/acsestwater.5c00551 · Cited by 6 · see citing works
Self-citationThe Rise of Open Source Models and Implications of Democratizing AI
Dey, S. (2025) · Computer · doi:10.1109/mc.2025.3548824 · Cited by 6 · see citing works
A conversational multi-agent AI system for automated plant phenotyping
Chen, F., Stogiannidis, I., Wood, A., Bueno, D., Williams, D., Macfarlane, F., Grieve, B., Wells, D. M., Atkinson, J. A., Hawkesford, M., et al. (2025) · Nature Communications · doi:10.1038/s41467-026-71090-y · Cited by 13 · see citing works
“Since PhenoAssistant leverages rapidly evolving AI technologies, this evaluation also offers a method for assessing new AI components (e.g. DeepSeek 43,44 ) as replacements for PhenoAssistant ’ s original modules.”
Supporting Sustainable Computing by Repurposing E-Waste Smartphones as Tiny Data Centers
Ngoy, P. M., Dar, F., Liyanage, M., Yin, Z., Norbisrath, U., Zuniga, A., Pestana, J., Radeta, M., Nurmi, P., & Flores, H. (2025) · IEEE Pervasive Computing · doi:10.1109/mprv.2025.3541558 · Cited by 6 · see citing works
EdgeAIGuard: Agentic LLMs for Minor Protection in Digital Spaces
Mujtaba, G., Khowaja, S. A., & Dev, K. (2025) · IEEE Internet of Things Journal · doi:10.1109/jiot.2025.3574961 · Cited by 6 · see citing works
“As mentioned in the algorithm and architectural workflow, we would leverage DeepSeek-7B (distilled version) [37], which is optimized for edge deployment without compromising performance.”
Confidence v.s. Critique: A Decomposition of Self-Correction Capability for LLMs
Yang, Z., Zhang, Y., Wang, Y., Xu, Z., Lin, J., & Sui, Z. (2025) · Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2025.acl-long.203 · Cited by 5 · see citing works
Self-citationDas, A., Bhattacharya, M., Abdelhameed, A. S., Lee, S., & Chakraborty, C. (2025) · Molecular Therapy — Nucleic Acids · doi:10.1016/j.omtn.2025.102546 · Cited by 5 · see citing works
Anchor Attention, Small Cache: Code Generation With Large Language Models
Zhang, X., Zhou, Y., Yang, G., Gall, H. C., & Chen, T. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3570680 · Cited by 6 · see citing works
“Inthefuture, we plan to extend the application of Anchor Attention to a broader spectrum of attention mechanisms, including but not limited to multi-query attention (MQA) [70], multi-head latent attention (MLA) [71], and grouped-query attention (GQA) [72].”
DeepSeek: implications for data science and management in the AI era
Xu, Z., Liu, S., Huang, W., Shi, J., Xu, F., Tang, X., & Lu, H. (2025) · Data Science and Management · doi:10.1016/j.dsm.2025.09.001 · Cited by 5 · see citing works
Self-citationSmall, Medium, and Large Language Models for Text-to-SQL
de Oliveira, A. R., Nascimento, E., Pinheiro, J., Avila, C. V. S., Coelho, G. M. C., Feijó, L., Izquierdo, Y. T., García, G. M., Leme, L. A. P. P., Lemos, M., et al. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-75872-0_15 · Cited by 5 · see citing works
Church, K. (2025) · International Journal of Lexicography · doi:10.1093/ijl/ecaf007 · Cited by 5 · see citing works
Comet: Accelerating Private Inference for Large Language Model by Predicting Activation Sparsity
Yan, G., Zhang, Y., Guo, Z., Zhao, L., Chen, X., Wang, C., Wang, W., Meng, D., & Hou, R. (2025) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp61157.2025.00182 · Cited by 5 · see citing works
Self-citationSafetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress?
Ren, R., Basart, S., Khoja, A., Gatti, A., Phan, L., Yin, X., Mazeika, M., Pan, A., Mukobi, G., Kim, R. H., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.21792 · Cited by 10 · see citing works
Khan, G., Sanford, C., & Trinh, C. T. (2025) · Biotechnology Advances · doi:10.1016/j.biotechadv.2025.108738 · Cited by 4 · see citing works
Zhang, Y.-H., Qiu, Q., Liu, X., Yu, X., Fu, D.-X., Liu, X.-Y., Wang, Z., Lin, H., Chen, Y., Yi, L., et al. (2026) · Journal of Optical Communications and Networking · doi:10.1364/jocn.576017 · Cited by 6 · see citing works
“Modern general LLMs such as GPT [36], Claude [57], Llama [58], Gemini [59], Mistral [60], Deepseek [61], Qwen [62], and Doubao [63] are pre-trained on vast general-purpose corpora, encompassing text from a wide range of domains.”
Self-citationEdge computing for IoT: Novel insights from a comparative analysis of access control models
Xue, T., Zhang, Y., Wang, Y., Wang, W., Li, S., & Zhang, H. (2025) · Computer Networks · doi:10.1016/j.comnet.2025.111468 · Cited by 4 · see citing works
Self-citationSchmidt‐Lebuhn, A. N., & Knerr, N. J. (2025) · PhytoKeys · doi:10.3897/phytokeys.261.158396 · Cited by 4 · see citing works
Proceedings of the Workshop on Edge and Mobile Foundation Models
(2024) · doi:10.1145/3662006 · Cited by 4 · see citing works
Cheng, Z., Zhou, L., Jiang, F., Wang, B., & Li, H. (2025) · Proceedings of the ACM on Web Conference 2025 · doi:10.1145/3696410.3714770 · Cited by 3 · see citing works
Self-citationMoEcho: Exploiting Side-Channel Attacks to Compromise User Privacy in Mixture-of-Experts LLMs
Ding, R., Xu, T., Shen, X., Ding, A. A., & Fei, Y. (2025) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3719027.3765174 · Cited by 4 · see citing works
GPT vs. Other Large Language Models for Topic Modeling: A Comprehensive Comparison
Meram, M. B., Kalkan, Ç., Çelikten, T., & Onan, A. (2025) · ICCK Transactions on Emerging Topics in Artificial Intelligence · doi:10.62762/tetai.2025.871572 · Cited by 4 · see citing works
Quantifying Variance in Evaluation Benchmarks
Madaan, L., Singh, A. K., Schaeffer, R., Poulton, A., Koyejo, S., Stenetorp, P., Narang, S., & Hupkes, D. (2024) · arXiv · doi:10.48550/arxiv.2406.10229 · Cited by 2 · see citing works
“…2023), Pythia {1, 1.4, 2.8, 6.9, 12}B (Biderman et al., 2023), Falcon {7, 40}B (Almazrouei et al., 2023), DeepSeek {7, 67}B (Bi et al., 2024), DeepSeek-MoE 16B (Bi et al., 2024), DeepSeek V2 (DeepSeek-AI, 2024), StableLM {1.6, 3, 7}B (StabilityAI, 2024), and MPT {7, 30}B (MosaicML NLP Team, 2023).”
InfluentialObservational Scaling Laws and the Predictability of Language Model Performance
Ruan, Y., Maddison, C. J., & Hashimoto, T. (2024) · arXiv · doi:10.48550/arxiv.2405.10938 · Cited by 3 · see citing works
DSciSum: Detailed summarization of long scientific documents
Liu, R., Mao, X.-L., & Huang, H. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113409 · Cited by 2 · see citing works
Large Language Model for Verilog Code Generation: Literature Review and the Road Ahead
Yang, G., Zheng, W., Liang, D., Hu, P., Yang, Y., Peng, S., Li, Z., Feng, J., Wei, X., Sun, K., et al. (2026) · ACM Computing Surveys · doi:10.1145/3841636 · Cited by 2 · see citing works
“Based on their adaptation for Verilog tasks, we categorize these LLMs into two main types: Base LLMs ( general-purpose models without Verilog-specific adaptation ) and IT LLMs ( models specifically Instruction-Tuned for Verilog code generation ).”
Animating the Past: Reconstruct Trilobite via Video Generation
Wu, X., Huang, Z., & Yu, C. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825790 · Cited by 1 · see citing works
“(1) Pretraining: This phase involves training an initial model π 0 on a large text corpus to optimize the prediction of the next token based on the preceding text [41]–[43].”
RSINS‐GS: Reconstruction From Single Image With Noise‐Added Strategy and 3D‐GS
Qian, S., Cheng, L., Li, P., & Xu, X. (2025) · IET Image Processing · doi:10.1049/ipr2.70082 · Cited by 1 · see citing works
A Large Language Model Evaluation Method for Legal Case Retrieval
Wang, Y., Zhao, X., Qi, X., Chen, B., Ma, C., & Xu, Y. (2025) · Data Intelligence · doi:10.3724/2096-7004.di.2025.0037 · Cited by 1 · see citing works
Self-citationHyperCRX 2.0: A Comprehensive and Automated Tool for Empowering GitHub Insights
Wang, Y., Wang, W., Xia, X., Tang, Y., Bi, F., Han, F., Peng, J., Zhao, S., & You, L. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse-companion66252.2025.00022 · Cited by 1 · see citing works
“OSS-GPT supports customizable configurations for various commonly used LLMs including those from the OpenAI [9] series, DeepSeek [10] series, Qwen [11] series, and locally deployed open source models running on Ollama (such as Llama [12]).”
Self-citationTowards enhanced LLM pretraining: Dynamic checkpoint merging via generation quality
Wang, Z., Liu, D., Li, C., Chu, D., Chen, W., Wang, B., & Sui, D. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103415 · Cited by 1 · see citing works
Self-citationCharacter-Level Adversarial Samples Generation Method Based on Dual-Layer Text Watermark
Wu, H., Li, X., Zhang, S., & Zhu, H. (2025) · IEEE Access · doi:10.1109/access.2025.3594727 · Cited by 1 · see citing works
“In recent years, with the widespread application of pre-trained language models in Chinese natural language processing tasks [1], [2], [3], the generation of textual adversarial samples has emerged as a critical area of research for enhancing model robustness.”
Yi, Z., Liang, Z., Xie, T., Fu, Y., & Liu, Y. (2025) · International Journal of Production Economics · doi:10.1016/j.ijpe.2025.109820 · Cited by 1 · see citing works
Self-citationMao, K., Hu, B., Lin, R., Li, Z., Lu, G., & Zhang, Z. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2025.1660912 · Cited by 4 · see citing works
Chen, Y., Ma, C., Li, Q., Yan, F., Chen, Y., Li, T., Ye, J., Hu, M., Lin, Y., Li, Y., et al. (2026) · Nature Cancer · doi:10.1038/s43018-026-01220-4 · Cited by 1 · see citing works
“To ensure visual dependency, we applied LLM-based filtering 41 , excluding questions answerable by at least three of four text-only LLMs (that is, GPT-4, InternLM2-Chat-7B 42 , Qwen-7B-Chat 43 and DeepSeek-7B-Chat 44 ) without images.”
Self-citationGuan, B., Lu, J., Tang, Y., & Long, R. (2024) · International Conference on Computing, Robotics and System Sciences (ICRSS) · doi:10.1109/icrss65752.2024.00027 · Cited by 1 · see citing works
“In the context of this study, we have opted to utilize Deepseek, a large language model, for sentiment analysis purposes [6].”
OneEncoder: a lightweight framework for efficient multimodal training
Faye, B., Azzag, H., Lebbah, M., & Bouchaffra, D. (2025) · Neural Computing and Applications · doi:10.1007/s00521-025-11435-8 · Cited by 1 · see citing works
Torezano, M. D. C. B. (2025) · ELUA Estudios de Lingüística Universidad de Alicante · doi:10.14198/elua.29420 · Cited by 1 · see citing works
A two-stage model for unified sentence- and document-level biomedical event extraction
Su, F., Zhang, Y., Jiao, P., Zhao, Z., Li, B., Li, F., & Ji, D. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.112001 · Cited by 1 · see citing works
Self-citationAn Efficient Finetuning Method for LLM generated text detection in Power Grid
Jiang, Y., Li, J., Zhang, X., Xu, W., Liang, Z., Yang, Y., Huang, K., & Bi, L. (2025) · IEEE/CIC International Conference on Communications in China (ICCC) · doi:10.1109/iccc65529.2025.11148942 · Cited by 1 · see citing works
“Nowadays, LLM-generated texts can be seen almost every-where, due to the effectiveness of GPT series of models [1], [16] and similar LLMs [2].”
Self-citationAi, Q., Zhan, J., & Liu, Y. (2025) · arXiv · doi:10.48550/arxiv.2501.02842
“The filtering methods usually employ neural scoring models and based on the credibility of the site [40, 41].”
Self-citationSMoA: Improving Multi-agent Large Language Models with Sparse Mixture-of-Agents
Li, D., Tan, Z., Qian, P., Li, Y., Chaudhary, K. S., Hu, L., & Shen, J. (2024) · arXiv · doi:10.48550/arxiv.2411.03284
Self-citationConditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models
Cheng, X., Zeng, W., Dai, D., Chen, Q., Wang, B., Xie, Z., Huang, K., Yu, X., Hao, Z., Zhang, H., et al. (2026) · Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2026.acl-long.226 · Cited by 3 · see citing works
Self-citationTraining Compute-Optimal Transformer Models: A Review of Scaling Laws, Allocation, and Practice
Memming, A. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7142719
Beyond Statistical Learning: Exact Learning Is Essential for General Intelligence
György, A., Lattimore, T., Lazić, N., & Szepesvári, C. (2025) · arXiv · doi:10.48550/arxiv.2506.23908
MPPQA: Structure-Aware Extractive Multi-span Question Answering for Procedural Documents
Zhou, B., Ren, H., Cai, Y., Lian, Z., Zhu, P., & Zeng, Y. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-9431-7_10
Liu, X., Yin, D., Zhang, C., Zhao, D., & Feng, Y. (2025) · Computational Linguistics · doi:10.1162/coli_a_00548
Self-citationFalconProtein: Finetuning Falcon Foundation Model for Protein Engineering
Yang, W., Shen, Y., Wang, Z., Zhao, R., Lu, Q., Liu, X., Liu, Y., Debbah, M. A., Wang, Y. G., & Wang, S. L. (2024) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm62325.2024.10822514
“5B [39],Chatglm-6B [40],Deepseek-llm-7B-chat [41], Falcon2-11B [10],BioMistral-7B [42],Meta-Llama-3.”
Self-citationRobust Visual Food Recognition for Enriching Nutrition Knowledge Bases
Ming, Z., Xie, Z., Zhang, C., Su, K., Yuan, C., & Chua, T. (2025) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2025.3542962
“For our implementation, we will utilize DeepSeek [35] and LLaMA [36] as examples of the LLMs in this approach.”
Self-citationLIGHT-TABLE-CHAIN: Simplifying and Enhancing Chain-Based Table Reasoning
Wang, H., Liu, W., Wang, Q., Xu, J., Nie, S., Liu, Y., & Shi, R. (2024) · IEEE International Conference on Computer and Communications (ICCC) · doi:10.1109/iccc62609.2024.10942246
“A. Experiment Settings 1) Baselines: We compare baseline methods under three models: GPT-4o-2024-08-06, DeepSeek-Chat [5], and LLaMA 3.1 8B [6].”
InfluentialSelf-citationLegal long-text case retrieval method based on multiscale feature fusion technology
Wang, Y., Qi, X., & Zhao, X. (2025) · International Conference on Computer Application and Information Security (ICCAIS) · doi:10.1117/12.3061478
Self-citationTEXT-TO-SQL ON REAL WORLD DATASETS
OLIVEIRA, A. R. D. (2024) · doi:10.17771/pucrio.acad.70127
A Method for Detecting Knowledge Conflicts in Chinese Intelligent Agent Interactions
Cheng, H., Gu, Y., Xi, M., Zhong, Q., & Liu, W. (2025) · Interdisciplinary Journal of Information Knowledge and Management · doi:10.28945/5497
Self-citationTraffic Flow Prediction Using DeepSeek Fine-Tuning and Dynamic Modeling
高, 畅. (2025) · Open Journal of Transportation Technologies · doi:10.12677/ojtt.2025.143034
Achieving Professional Translation in the Military Field Through Fine-Tuning LLM
Tao, Y., Yang, R., Meng, F., Liu, Y., & Zhuo, W. (2025) · International Seminar on Artificial Intelligence, Networking and Information Technology (AINIT) · doi:10.1109/ainit65432.2025.11035793
Self-citationNeural Classification of Argument Elements and Styles in Arabic Competitive Debates
Al-Zawqari, A., Ahmed, M., Al-Sharafi, A. G., Khader, M., Safa, A., & Vandersteen, G. (2025) · IEEE Access · doi:10.1109/access.2025.3585299
“We evaluate two open-source LLMs: DeepSeek-7B [30] (transformer-based) and Falcon3-Mamba-7B [31] (state-space architecture).”
From Screen to Scene: An In-Depth Forensic Analysis on DeepseekPreprint
Gong, Y., Tyagi, S., Mahindra, V., & Karabiyik, Ü. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5362681
Tang, D., & Miškov-Živanov, N. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.08.06.668950
“To study the impact of model size, we also fine-tune three LLMs on the same data: DeepSeek-7B 28 and Llama3.”
Achieving consistency in FedSAM using local adaptive distillation on sports image classification
Zhen, K., wu, J., Park, J., Shao, R., Zhang, X., & Yu, S. (2025) · PLoS ONE · doi:10.1371/journal.pone.0333210
Self-citationPrompting the Mind: EEG-to-Text Translation with Multimodal LLMs and Semantic Control
Al-Radhi, M. S., Shurid, S. M., & Németh, G. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-07956-5_4
- Authors
- DeepSeek-AI, :, Bi, X., Chen, D., Chen, G., Chen, S., Dai, D., Deng, C., Ding, H., Dong, K., Du, Q., Fu, Z., Gao, H., Gao, K., Gao, W., Ge, R., Guan, K., Guo, D., Guo, J., Hao, G., Hao, Z., He, Y., Hu, W., Huang, P., Li, E., Li, G., Li, J., Li, Y., Li, Y. K., Liang, W., Lin, F., Liu, A. X., Liu, B., Liu, W., Liu, X., Liu, X., Liu, Y., Lu, H., Lu, S., Luo, F., Ma, S., Nie, X., Pei, T., Piao, Y., Qiu, J., Qu, H., Ren, T., Ren, Z., Ruan, C., Sha, Z., Shao, Z., Song, J., Su, X., Sun, J., Sun, Y., Tang, M., Wang, B., Wang, P., Wang, S., Wang, Y., Wang, Y., Wu, T., Wu, Y., Xie, X., Xie, Z., Xie, Z., Xiong, Y., Xu, H., Xu, R. X., Xu, Y., Yang, D., You, Y., Yu, S., Yu, X., Zhang, B., Zhang, H., Zhang, L., Zhang, L., Zhang, M., Zhang, M., Zhang, W., Zhang, Y., Zhao, C., Zhao, Y., Zhou, S., Zhou, S., Zhu, Q., & Zou, Y.
- ORCID
- Chen, Deli
- Dai, Damai
- Deng, Chengqi
- Fu, Zhe
- Gao, Huazuo
- Gao, Wenjun
- Guo, Daya
- Hao, Zhewen
- He, Ying
- Huang, Panpan
- Li, Guowei
- Li, Jiashi
- Li, Y. K.
- Liu, A. X.
- Liu, Bo
- Liu, Wen
- Liu, Xiaodong
- Liu, Xin
- Liu, Yiyuan
- Lu, Haoyu
- Luo, Fuli
- Ma, Shirong
- Qiu, Junjie
- Qu, Hui
- Ren, Tongzheng
- Ren, Zehui
- Shao, Zhihong
- Song, Junxiao
- Tang, Minghui
- Wang, Peiyi
- Wang, Shiyu
- Wang, Yongji
- Wu, Tong
- Wu, Y.
- Xie, Xin
- Xie, Ziwei
- Xu, Yanhong
- Yu, Xingkai
- Zhang, B.
- Zhang, Haowei
- Zhang, Liyue
- Zhang, Mingchuan
- Zhang, Minghua
- Zhang, Wentao
- Zhao, Chenggang
- Zhao, Yao
- Zhou, Shangyan
- Zhu, Qihao
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2401.02954
- OpenAlex
- W4390690464
- Cited by
- 99
- References
- 71
- Citations by year
- 2026: 24 · 2025: 59 · 2024: 16
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- open-source large language models, scaling laws, long-termism, pre-training dataset, supervised fine-tuning, direct preference optimization, code generation, mathematical reasoning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
