Phi-4 Technical Report
Open accessAbstract
We present phi-4, a 14-billion parameter language model developed with a training recipe that is centrally focused on data quality. Unlike most language models, where pre-training is based primarily on organic data sources such as web content or code, phi-4 strategically incorporates synthetic data throughout the training process. While previous models in the Phi family largely distill the capabilities of a teacher model (specifically GPT-4), phi-4 substantially surpasses its teacher model on STEM-focused QA capabilities, giving evidence that our data-generation and post-training techniques go beyond distillation. Despite minimal changes to the phi-3 architecture, phi-4 achieves strong performance relative to its size -- especially on reasoning-focused benchmarks -- due to improved data, training curriculum, and innovations in the post-training scheme.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Synthetic data has emerged as a potential remedy for data scarcity in LLM pre-training.
“making synthetic data a potential remedy”
From Scaling Data-Constrained Language Models with Synthetic Data (2026) · cites this paper as Abdin et al., 2024
Section 2.1 says synthetic data is becoming a substantial component of pretraining, and Section 3.1 reports that reusing synthetic data was more beneficial than adding fresh web tokens in some experiments. These passages support its potential as an alternative source of training data, but they do not explicitly frame it as a remedy for data scarcity; Section 3.1 also notes limitations of synthetic-only training. The supplied copy gives no page numbers. (LLM, conf=0.82)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughly
Yen, H., Gao, T., Hou, M., Ding, K., Fleischer, D., Izsak, P., Wasserblat, M., & Chen, D. (2024) · arXiv · Cited by 2 · cited 1× · see citing works
“We report the performance of phi-4 and other models on the tasks we selected from the HELMET [YGH + 24] evaluation suite in Table 6 and outline our evaluation methods below.”
Flagged first, then by citations
Research works printed but not linked: 12
- Marah Abdin, Sam Jacobs Ade, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219, 2024. · arXiv
- Meta AI. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2023. · arXiv
- Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evalu- ating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. · arXiv
- Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Gustavo de Rosa Piero Kauffmann, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang, Sébastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Yin Tat Lee, and Yuanzhi Li. Textbooks are all you need. arXiv preprint arXiv:2306.11644, 2023. · arXiv
- Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020. · arXiv
- Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurIPS, 2021.
- Emman Haider, Daniel Perez-Becker, Thomas Portet, Piyush Madan, Amit Garg, Atabak Ashfaq, David Majercak, Wen Wen, Dongwoo Kim, Ziyi Yang, et al. Phi-3 safety post-training: Aligning language models with a "break-fix"cycle. arXiv preprint arXiv:2407.13833, 2024. · arXiv
- Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. Textbooks are all you need II: phi-1.5 technical report. arXiv preprint arXiv:2309.05463, 2023. · arXiv
- Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. arXiv preprint arXiv:2305.01210, 2023. · arXiv
- Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, 2017.
- Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911, 2023. · arXiv
Cited by
Sorted by citations
Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al. (2025) · arXiv · doi:10.48550/arxiv.2505.09388 · Cited by 111 · see citing works
Self-citationLLMs Get Lost In Multi-Turn Conversation
Laban, P., Hayashi, H., Zhou, Y., & Neville, J. (2025) · arXiv · Cited by 10 · see citing works
“3-70B-Instruct, Llama 4 Scout) [23], AI2 OLMo-2-13B [56], Microsoft Phi-4 [1], Deepseek-R1 [24], and Cohere Command-A [15].”
FactualRAG: a RAG-Powered framework for LLM fact-checking via IFCN-Accredited sources
Chang, L.-J., & Lai, C.-M. (2026) · Journal of Intelligent Information Systems · doi:10.1007/s10844-026-01041-y · Cited by 5 · see citing works
Quantification of Large Language Model Distillation
Lee, S., Zhou, J., Ao, C., Li, K., Du, X., He, S., Wu, H., Liu, T., Liu, J., Alinejad-Rokny, H., et al. (2025) · Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2025.acl-long.248 · Cited by 4 · see citing works
On the Impacts of Contexts on Repository-Level Code Generation
Hai, N. L., Nguyen, D. M., & Bui, N. D. Q. (2024) · arXiv · doi:10.48550/arxiv.2406.11927 · Cited by 3 · see citing works
Back into Plato's Cave: Examining Cross-modal Representational Convergence at Scale
Koepke, A. S., Zverev, D., Ginosar, S., & Efros, A. A. (2026) · arXiv · doi:10.48550/arxiv.2604.18572
UltraEdit: Training-, Subject-, and Memory-Free Lifelong Editing in Language Models
Gu, X., Huang, Z., Gu, J.-C., & Zhang, K. (2025) · arXiv · doi:10.48550/arxiv.2505.14679
Scaling Data-Constrained Language Models with Synthetic Data
Kiyomaru, H., Oda, Y., Kodama, T., Liu, C., & Kawahara, D. (2026) · Findings of the Association for Computational Linguistics: EACL 2026 · doi:10.18653/v1/2026.findings-eacl.52
Cartridges: Lightweight and general-purpose long context representations via self-study
Eyuboglu, S., Ehrlich, R., Arora, S., Guha, N., Zinsley, D., Liu, E., Tennien, W., Rudra, A., Zou, J., Mirhoseini, A., et al. (2025) · arXiv · doi:10.48550/arxiv.2506.06266
LittleBit: Ultra Low-Bit Quantization via Latent Factorization
Lee, B., Kim, D., You, Y., & Kim, Y. (2025) · arXiv · doi:10.48550/arxiv.2506.13771
“Evaluation Setup We evaluate LittleBit on diverse LLM families including Llama [48], Llama2 [49], Llama3 [50], OPT [51], Phi-4 [52], and Qwen [53], spanning sizes from 1.3B to 32B parameters.”
- Authors
- Abdin, M., Aneja, J., Behl, H., Bubeck, S., Eldan, R., Gunasekar, S., Harrison, M., Hewett, R. J., Javaheripi, M., Kauffmann, P., Lee, J. R., Lee, Y. T., Li, Y., Liu, W., Mendes, C. C. T., Nguyen, A., Price, E., de Rosa, G., Saarikivi, O., Salim, A., Shah, S., Wang, X., Ward, R., Wu, Y., Yu, D., Zhang, C., & Zhang, Y.
- Affiliations
- Microsoft (United States) (US)
- ORCID
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2412.08905
- OpenAlex
- W4405354744
- Cited by
- 31
- References
- 1
- Citations by year
- 2026: 15 · 2025: 16
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- Microsoft Research
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- Phi-4, large language models, synthetic data, data quality, knowledge distillation, GPT-4, post-training, training curriculum, reasoning benchmarks, teacher-student models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
