CoChat
Sign Up
PreprintCited by 27Software Engineering ResearchTopic ModelingTeaching and Learning Programming

Magicoder: Empowering Code Generation with OSS-Instruct

Open access
Wei, Yuxiang, Wang, Zhe, Liu, Jiawei, Ding, Yifeng, Zhang, Lingming·2023·arXiv
DOI 10.48550/arxiv.2312.02120arXiv 2312.02120
Open this paper in CoChat — free accountOpen PDF

Abstract

We introduce Magicoder, a series of fully open-source (code, weights, and data) Large Language Models (LLMs) for code that significantly closes the gap with top code models while having no more than 7B parameters. Magicoder models are trained on 75K synthetic instruction data using OSS-Instruct, a novel approach to enlightening LLMs with open-source code snippets to generate diverse instruction data for code. Our main motivation is to mitigate the inherent bias of the synthetic data generated by LLMs through the wealth of open-source references for the production of more realistic and controllable data. The orthogonality of OSS-Instruct and other data generation methods like Evol-Instruct further enables us to build an enhanced MagicoderS. Both Magicoder and MagicoderS substantially outperform state-of-the-art code models with similar or even larger sizes on a wide range of coding benchmarks. Notably, MagicoderS-CL-7B based on CodeLlama even surpasses the prominent ChatGPT on HumanEval+ (66.5 vs. 65.9 in pass@1 ). Overall, OSS-Instruct opens a new direction for crafting diverse synthetic instruction data for code using abundant open-source references.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2312.02120

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • MCEVAL-INSTRUCT contains 110K samples, comprised of created question-answer pairs and open-source collection.

    “The resulting dataset, MCEVAL-INSTRUCT (110K samples), is comprised of created question-answer pairs and open-source collection (Wei et al., 2023).”

    From McEval: Massively Multilingual Code Evaluation (2024) · cites this paper as Wei et al., 2023

    The “Similarity with HumanEval” section identifies the paper’s OSS-INSTRUCT dataset as 75K samples and assigns the 110K figure to a different dataset, evol-codealpaca-v1. The Introduction says OSS-INSTRUCT generates coding problems from open-source code snippets; our copy does not identify a dataset called MCEVAL-INSTRUCT or describe the claimed composition. (LLM, conf=0.88)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

No citation lists for this paper

No citation lists for this paper yet — checked again daily.

Cited by 27

Research works printed but not linked: 29

  1. Austin, J., Odena, A., Nye, M. I., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C. J., Terry, M., Le, Q. V., and Sutton, C. Program synthesis with large language models. CoRR, abs/2108.07732, 2021. URL https: //arxiv.org/abs/2108.07732. · arXiv
  2. Bouzenia, I., Devanbu, P., and Pradel, M. Repairagent: An autonomous, llm-based agent for program repair. arXiv preprint arXiv:2403.17134, 2024. · arXiv
  3. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901.
  4. Curran Associates, Inc., 2020. URL https://proceedings.neurips. cc/paper_files/paper/2020/file/ 1457c0d6bfcb4967418bfb8ac142f64a-Paper. pdf.
  5. Cambronero, J., Gulwani, S., Le, V., Perelman, D., Rad- hakrishna, A., Simon, C., and Tiwari, A. Flashfill++: Scaling programming by example by cutting to the chase. Proc. ACM Program. Lang., 7(POPL), jan 2023. doi: 10.1145/3571226. URL https://doi.org/10. 1145/3571226. · DOI
  6. Chen, M., Tworek, J., Jun, H., Yuan, Q., de Oliveira Pinto, H. P., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A., Puri, R., Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., Ryder, N., Pavlov, M., Power, A., Kaiser, L., Bavar- ian, M., Winter, C., Tillet, P., Such, F. P., Cummings, D., Plappert, M., Chantzis, F., Barnes, E., Herbert-Voss, A., Guss, W. H., Nichol, A., Paino, A., Tezak, N., Tang, J., Babuschkin, I., Balaji, S., Jain, S., Saunders, W., Hesse, C., Carr, A. N., Leike, J., Achiam, J., Misra, V., Morikawa, E., Radford, A., Knight, M., Brundage, M., Murati, M., Mayer, K., Welinder, P., McGrew, B., Amodei, D., McCandlish, S., Sutskever, I., and Zaremba, W. Evaluating large language models trained on code, 2021.
  7. Ding, Y., Wang, Z., Ahmad, W. U., Ding, H., Tan, M., Jain, N., Ramanathan, M. K., Nallapati, R., Bhatia, P., Roth, D., and Xiang, B. Crosscodeeval: A diverse and multilingual benchmark for cross-file code comple- tion. In Thirty-seventh Conference on Neural Informa- tion Processing Systems Datasets and Benchmarks Track, 2023. URL https://openreview.net/forum? id=wgDcbBMSfh.
  8. Feng, Y., Martins, R., Bastani, O., and Dillig, I. Program synthesis using conflict-driven learning. SIGPLAN Not., 53(4):420-435, jun 2018. ISSN 0362-1340. doi: 10. 1145/3296979.3192382. URL https://doi.org/ 10.1145/3296979.3192382. · DOI
  9. Fried, D., Aghajanyan, A., Lin, J., Wang, S., Wallace, E., Shi, F., Zhong, R., Yih, S., Zettlemoyer, L., and Lewis, M. Incoder: A generative model for code infilling and synthesis. In The Eleventh International Confer- ence on Learning Representations, 2023. URL https: //openreview.net/forum?id=hQwb-lbM6EL.
  10. Gulwani, S., Polozov, O., and Singh, R. Program syn- thesis. Foundations and Trends® in Programming Lan- guages, 4(1-2):1-119, 2017. ISSN 2325-1107. doi: 10.1561/2500000010. URL http://dx.doi.org/ 10.1561/2500000010. · DOI
  11. Honovich, O., Scialom, T., Levy, O., and Schick, T. Unnat- ural instructions: Tuning language models with (almost) no human labor. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Proceedings of the 61st Annual Meet- ing of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pp. 14409-14428, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.acl-long.806. URL https: //aclanthology.org/2023.acl-long.806. · DOI
  12. Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., and Narasimhan, K. Swe-bench: Can language mod- els resolve real-world github issues?, 2023.
  13. Lemieux, C., Inala, J. P., Lahiri, S. K., and Sen, S. Co- damosa: Escaping coverage plateaus in test genera- tion with pre-trained large language models. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), pp. 919-931. IEEE, 2023.
  14. Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., Hubert, T., Choy, P., de Masson d'Autume, C., Babuschkin, I., Chen, X., Huang, P.-S., Welbl, J., Gowal, S., Cherepanov, A., Molloy, J., Mankowitz, D. J., Sutherland Robson, E., Kohli, P., de Freitas, N., Kavukcuoglu, K., and Vinyals, O. Competition- level code generation with alphacode. Science, 378 (6624):1092-1097, December 2022. ISSN 1095-9203. doi: 10.1126/science.abq1158. URL http://dx.doi. org/10.1126/science.abq1158. · DOI · Paper page
  15. Liu, J., Peng, J., Wang, Y., and Zhang, L. Neuri: Di- versifying dnn generation via inductive rule inference. In Proceedings of the 31st ACM Joint European Soft- ware Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2023, pp. 657-669, New York, NY, USA, 2023a. Associa- tion for Computing Machinery. ISBN 9798400703270. doi: 10.1145/3611643.3616337. URL https://doi. org/10.1145/3611643.3616337. · DOI
  16. Liu, J., Xia, C. S., Wang, Y., and Zhang, L. Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation. In Thirty- seventh Conference on Neural Information Processing Systems, 2023b. URL https://openreview.net/ forum?id=1qvx610Cu7.
  17. Luo, Z., Xu, C., Zhao, P., Sun, Q., Geng, X., Hu, W., Tao, C., Ma, J., Lin, Q., and Jiang, D. Wizardcoder: Empowering code large language models with evol-instruct. arXiv preprint arXiv:2306.08568, 2023a. · arXiv
  18. Nijkamp, E., Pang, B., Hayashi, H., Tu, L., Wang, H., Zhou, Y., Savarese, S., and Xiong, C. Codegen: An open large language model for code with multi-turn pro- gram synthesis. In The Eleventh International Confer- ence on Learning Representations, 2023. URL https: //openreview.net/forum?id=iaYcJKpY2B_.
  19. Olausson, T. X., Inala, J. P., Wang, C., Gao, J., and Solar-Lezama, A. Is self-repair a silver bullet for code generation? In The Twelfth International Conference on Learning Representations, 2024. URL https:// openreview.net/forum?id=y0GJXRungR.
  20. Schäfer, M., Nadi, S., Eghbali, A., and Tip, F. An empirical evaluation of using large language models for automated unit test generation. IEEE Transactions on Software En- gineering, 2023.
  21. SPARCK JONES, K. A statistical interpretation of term specificity and its application in retrieval. 28(1):11-21, 2023/11/30 1972. doi: 10.1108/eb026526. URL https: //doi.org/10.1108/eb026526. · DOI
  22. Su, H., Shi, W., Kasai, J., Wang, Y., Hu, Y., Ostendorf, M., Yih, W.-t., Smith, N. A., Zettlemoyer, L., and Yu, T. One embedder, any task: Instruction-finetuned text em- beddings. 2022. URL https://arxiv.org/abs/ 2212.09741.
  23. Wang, X., Dillig, I., and Singh, R. Program synthesis using abstraction refinement. Proc. ACM Program. Lang., 2 (POPL), dec 2017. doi: 10.1145/3158151. URL https: //doi.org/10.1145/3158151. · DOI
  24. Wang, Y., Wang, W., Joty, S., and Hoi, S. C. CodeT5: Identifier-aware unified pre-trained encoder-decoder mod- els for code understanding and generation. In Moens, M.-F., Huang, X., Specia, L., and Yih, S. W.-t. (eds.), Pro- ceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 8696-8708, On- line and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.emnlp-main.685. URL https:// aclanthology.org/2021.emnlp-main.685. · DOI · Paper page
  25. Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Pro- ceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 13484-13508, Toronto, Canada, July 2023a. Associ- ation for Computational Linguistics. doi: 10.18653/v1/ 2023.acl-long.754. URL https://aclanthology. org/2023.acl-long.754. · DOI · Paper page
  26. Xia, C. S. and Zhang, L. Keep the conversation going: Fixing 162 out of 337 bugs for $0.42 each using chatgpt. arXiv preprint arXiv:2304.00385, 2023. · arXiv
  27. Xia, C. S., Wei, Y., and Zhang, L. Automated program repair in the era of large pre-trained language models. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), pp. 1482-1494, 2023b. doi: 10.1109/ICSE48619.2023.00129. · DOI · Paper page
  28. Xu, C., Sun, Q., Zheng, K., Geng, X., Zhao, P., Feng, J., Tao, C., and Jiang, D. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.12244, 2023. · arXiv
  29. Yuan, Z., Lou, Y., Liu, M., Ding, S., Wang, K., Chen, Y., and Peng, X. No more manual tests? evaluating and improving chatgpt for unit test generation. arXiv preprint arXiv:2305.04207, 2023. · arXiv

Authors
Wei, Y., Wang, Z., Liu, J., Ding, Y., & Zhang, L.
Affiliations
University of Illinois Urbana-Champaign (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2312.02120
arXiv
2312.02120
OpenAlex
W4389364446
Cited by
27
References
0
Citations by year
2026: 7 · 2025: 7 · 2024: 13
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
National Science Foundation (2131943)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
Keywords
code large language models, CodeLlama, HumanEval

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review