VeriGen: A Large Language Model for Verilog Code Generation
Open accessThe DOI matches its Crossref record; 7 of the 7 references it cites were found in a registry or an index.
Abstract
In this study, we explore the capability of Large Language Models (LLMs) to automate hardware design by automatically completing partial Verilog code, a common language for designing and modeling digital systems. We fine-tune pre-existing LLMs on Verilog datasets compiled from GitHub and Verilog textbooks. We evaluate the functional correctness of the generated Verilog code using a specially designed test suite, featuring a custom problem set and testing benches. Here, our fine-tuned open-source CodeGen-16B model outperforms the commercial state-of-the-art GPT-3.5-turbo model with a 1.1% overall increase. Upon testing with a more diverse and complex problem set, we find that the fine-tuned model shows competitive performance against state-of-the-art gpt-3.5-turbo, excelling in certain scenarios. Notably, it demonstrates a 41% improvement in generating syntactically correct Verilog code across various problem categories compared to its pre-trained counterpart, highlighting the potential of smaller, in-house LLMs in hardware design automation. We release our training/evaluation scripts and LLM checkpoints as open-source contributions.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Rasley, J., Rajbhandari, S., Ruwase, O., & He, Y. (2020) · Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3394486.3406703 · Cited by 823 · cited 1×
“We use model and data parallelism and strategies for sharding the optimizer states across GPUs similar to DeepSpeed5 [22] and ZeRO [23].”
MethodCodeSearchNet Challenge: Evaluating the State of Semantic Code Search
Husain, H., Wu, H.-H., Gazit, T., Allamanis, M., & Brockschmidt, M. (2019) · arXiv · doi:10.48550/arxiv.1909.09436 · Cited by 483 · cited 1×
“Despite the lack of open datasets for Verilog-focused LLM training, the promising outcomes from applying LLMs in programming, and their further potential shown in multi-language code repositories like CodeSearchNet [17], and large-scale LLMs such as GPT-3.”
BackgroundAsleep at the Keyboard? Assessing the Security of GitHub Copilot’s Code Contributions
Pearce, H., Ahmad, B., Tan, B., Dolan-Gavitt, B., & Karri, R. (2022) · 2022 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp46214.2022.9833571 · Cited by 476
Self-citation(2021) · doi:10.1145/3458817.3476205 · Cited by 257 · cited 2×
“We set the training hyperparameters to their defaults as recommended for the ZeRO-3 [26] optimizer state implementation.”
MethodTowards Human-Bot Collaborative Software Architecting with ChatGPT
Ahmad, A., Waseem, M., Liang, P., Fahmideh, M., Aktar, M. S., & Mikkonen, T. (2023) · Proceedings of the 27th International Conference on Evaluation and Assessment in Software Engineering · doi:10.1145/3593434.3593468 · Cited by 168 · cited 1×
“While considerable effort has been undertaken which explores the training and utilization of language models for writing software [1, 5, 14] as well as in their down-stream effects (e.”
BackgroundLost at C: A User Study on the Security Implications of Large Language Model Code Assistants
Sandoval, G., Pearce, H., Nys, T., Karri, R., Garg, S., & Dolan-Gavitt, B. (2022) · arXiv · doi:10.48550/arxiv.2208.09727 · Cited by 52
Self-citation
Cited by
Large Language Models for Software Engineering: A Systematic Literature Review
Hou, X., Zhao, Y., Liu, Y., Yang, Z., Wang, K., Li, L., Luo, X., Lo, D., Grundy, J., & Wang, H. (2023) · arXiv · doi:10.48550/arxiv.2308.10620 · Cited by 114
Chang, K., Wang, K., Yang, N., Wang, Y., Jin, D., Zhu, W., Chen, Z., Li, C., Yan, H., Zhou, Y., et al. (2024) · Proceedings of the 61st ACM/IEEE Design Automation Conference · doi:10.1145/3649329.3657356 · Cited by 60
Fight Fire with Fire: How Much Can We Trust ChatGPT on Source Code-Related Tasks?
Yu, X., Liu, L., Hu, X., Keung, J. W., Liu, J., & Xia, X. (2024) · arXiv · doi:10.48550/arxiv.2405.12641 · Cited by 29
Hu, X., Niu, F., Chen, J., Zhou, X., Zhang, J., He, J., Xia, X., & Lo, D. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3786771 · Cited by 15
Metadata
- Authors
- Thakur, S., Ahmad, B., Pearce, H., Tan, B., Dolan-Gavitt, B., Karri, R., & Garg, S.
- Venue
- ACM Transactions on Design Automation of Electronic Systems
- Published
- 2024-04-22
- Volume
- 29
- Issue
- 3
- Pages
- 1-31
- ISSN
- 1084-4309 (ISSN-L)
- 1557-7309
- OpenAlex
- W4391681217
- Cited by
- 228
- References
- 7
- FWCI
- 86.49
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.