CoChat
Sign Up
PreprintCited by 1Model-Driven Software Engineering TechniquesSoftware Engineering ResearchTopic Modeling

On Codex Prompt Engineering for OCL Generation: An Empirical Study

Open access
Abukhalaf, Seif, Hamdaqa, Mohammad, Khomh, Foutse·2023·arXiv
DOI 10.48550/arxiv.2303.16244arXiv 2303.16244
Open this paper in CoChat — free accountOpen PDF

Abstract

The Object Constraint Language (OCL) is a declarative language that adds constraints and object query expressions to MOF models. Despite its potential to provide precision and conciseness to UML models, the unfamiliar syntax of OCL has hindered its adoption. Recent advancements in LLMs, such as GPT-3, have shown their capability in many NLP tasks, including semantic parsing and text generation. Codex, a GPT-3 descendant, has been fine-tuned on publicly available code from GitHub and can generate code in many programming languages. We investigate the reliability of OCL constraints generated by Codex from natural language specifications. To achieve this, we compiled a dataset of 15 UML models and 168 specifications and crafted a prompt template with slots to populate with UML information and the target task, using both zero- and few-shot learning methods. By measuring the syntactic validity and execution accuracy metrics of the generated OCL constraints, we found that enriching the prompts with UML information and enabling few-shot learning increases the reliability of the generated OCL constraints. Furthermore, the results reveal a close similarity based on sentence embedding between the generated OCL constraints and the human-written ones in the ground truth, implying a level of clarity and understandability in the generated OCL constraints by Codex.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2303.16244

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • We selected five programming languages, guided by programming language popularity while also covering different programming paradigms.

    “To mitigate these threats, we selected five PLs, guided by PL popularity [21] while also covering different programming paradigms.”

    From Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code (2024) · cites this paper as [21]

    The Code generation and Background and Terminology sections discuss several programming languages, but neither addresses their popularity or uses popularity to guide language selection. Searches for programming-language popularity and selection of five languages across paradigms found no supporting passage in our copy. The citation is attached specifically to the popularity rationale. (LLM, conf=0.72)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 22

  1. A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann et al., "Palm: Scaling language modeling with pathways," arXiv preprint arXiv:2204.02311, 2022. · arXiv
  2. Y. Wang, W. Wang, S. Joty, and S. C. Hoi, "Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation," arXiv preprint arXiv:2109.00859, 2021. · arXiv
  3. M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman et al., "Evaluating large language models trained on code," arXiv preprint arXiv:2107.03374, 2021. · arXiv
  4. P. Liu, W. Yuan, J. Fu, Z. Jiang, H. Hayashi, and G. Neubig, "Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language processing," arXiv preprint arXiv:2107.13586, 2021. · arXiv
  5. A. R. Da Silva, "Model-driven engineering: A survey supported by the unified conceptual model," Computer Languages, Systems & Structures, vol. 43, pp. 139-155, 2015.
  6. S. Salemi, A. Selamat, and M. Penhaker, "A model transformation framework to increase ocl usability," Journal of King Saud University - Computer and Information Sciences, vol. 28, 11 2015.
  7. G. Poesia, O. Polozov, V. Le, A. Tiwari, G. Soares, C. Meek, and S. Gulwani, "Synchromesh: Reliable code generation from pre-trained language models," arXiv preprint arXiv:2201.11227, 2022. · arXiv
  8. C. Yu, G. Yang, X. Chen, K. Liu, and Y. Zhou, "Bashexplainer: Retrieval-augmented bash code comment generation based on fine-tuned codebert," 10 2022, pp. 82-93.
  9. J. Cabot, D. Delgado, and L. Burgueño, "Combining ocl and natural language: a call for a community effort," 11 2022, pp. 908-912.
  10. I. Bajwa, M. Lee, and B. Bordbar, "Translating natural language constraints to ocl," Journal of King Saud University -Computer and Information Sciences, vol. 24, p. 117-128, 07 2012.
  11. M. Richters and M. Gogolla, "Validating uml models and ocl con- straints," in International Conference on the Unified Modeling Lan- guage, 2000.
  12. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al., "Language mod- els are few-shot learners," Advances in neural information processing systems, vol. 33, pp. 1877-1901, 2020.
  13. N. Nguyen and S. Nadi, "An empirical evaluation of github copilot's code suggestions," in Proceedings of the 19th International Conference on Mining Software Repositories, 2022, pp. 1-5.
  14. J. Cabot and M. Gogolla, "Object constraint language (ocl): A definitive guide," vol. 7320, 06 2012, pp. 58-90.
  15. J. Noten, J. G. Mengerink, and A. Serebrenik, "A data set of ocl expres- sions on github," in 2017 IEEE/ACM 14th International Conference on Mining Software Repositories (MSR), 2017, pp. 531-534.
  16. J. G. Mengerink, J. Noten, and A. Serebrenik, "Empowering ocl re- search: a large-scale corpus of open-source data from github," Empirical Software Engineering, vol. 24, no. 3, pp. 1574-1609, 2019.
  17. C. Ashbacher, "The object constraint language second edition, getting your models ready for mda, by jos warmer and anneke kleppe." The Journal of Object Technology, vol. 2, p. 139, 2003.
  18. I. Bajwa, "A natural language processing approach to generate sbvr and ocl," Ph.D. dissertation, 12 2012.
  19. J. Cohen, "A coefficient of agreement for nominal scales," Educational and psychological measurement, vol. 20, no. 1, pp. 37-46, 1960.
  20. M. Gogolla, F. Büttner, and M. Richters, "Use: A uml-based specification environment for validating uml and ocl," Science of Computer Programming, vol. 69, no. 1, pp. 27-34, 2007, special issue on Experimental Software and Toolkits. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0167642307001608
  21. W. Wang, F. Wei, L. Dong, H. Bao, N. Yang, and M. Zhou, "Minilm: Deep self-attention distillation for task-agnostic compression of pre- trained transformers," CoRR, vol. abs/2002.10957, 2020. [Online]. Available: https://arxiv.org/abs/2002.10957 · arXiv
  22. N. Reimers and I. Gurevych, "Sentence-bert: Sentence embeddings using siamese bert-networks," in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019. [Online]. Available: https: //arxiv.org/abs/1908.10084

Cited by

Sorted by citations

Authors
Abukhalaf, S., Hamdaqa, M., & Khomh, F.
Affiliations
Polytechnique Montréal (CA)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2303.16244
arXiv
2303.16244
OpenAlex
W4361806156
Cited by
0
References
0
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
  • arXiv (Cornell University)
Funders
  • Canadian Institute for Advanced Research
  • Natural Sciences and Engineering Research Council of Canada
Topics
  • Physical Sciences › Computer Science › Software › Model-Driven Software Engineering Techniques
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
Object Constraint Language, codices, large language models, prompt engineering, UML models, few-shot learning, zero-shot learning, natural language specifications, execution accuracy, sentence embeddings, model-driven engineering

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review