LLM-Supported Natural Language to Bash Translation
Open accessAbstract
Finnian Westenfelder, Erik Hemberg, Stephen Moskal, Una-May O’Reilly, Silviu Chiricescu. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2025.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Other benchmarks measure narrow aspects of the command line interface, like optimizing shell scripts, configuring software environments, or translating natural language to Bash commands.
“Other benchmarks measure narrow aspects of the command line interface, like optimizing shell scripts (Lamprou et al., 2025), configuring software environments (Eliseeva et al., 2025), or translating natural language to Bash commands (Westenfelder et al., 2025).”
From Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces (2026) · cites this paper as Westenfelder et al., 2025
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 35
- Mayank Agarwal, Tathagata Chakraborti, Quchen Fu, David Gros, Xi Victoria Lin, Jaron Maene, Kartik Ta- lamadupula, Zhongwei Teng, and Jules White. 2021. Neurips 2020 nlc2cmd competition: Translating nat- ural language to bash commands. In Proceedings of the NeurIPS 2020 Competition and Demonstra- tion Track, volume 133 of Proceedings of Machine Learning Research, pages 302-324.
- PMLR.
- Pooja Aggarwal, Oishik Chatterjee, Ting Dai, Prateeti Mohapatra, Brent Paulovicks, Brad Blancett, and Arthur De Magalhaes. 2024. CodeSift: An LLM- Based Reference-Less Framework for Automatic Code Validation . In 2024 IEEE 17th International Conference on Cloud Computing (CLOUD), pages 404-410, Los Alamitos, CA, USA. IEEE Computer Society. · DOI
- Duarte Alves, Nuno Guerreiro, João Alves, José Pom- bal, Ricardo Rei, José de Souza, Pierre Colombo, and Andre Martins. 2023. Steering large language models for machine translation with finetuning and in-context learning. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 11127-11148, Singapore. Association for Computa- tional Linguistics. · DOI
- Shikhar Bharadwaj, Shirish Shevade, and Marine Carpuat. 2022. Efficient constituency tree based en- coding for natural language to bash translation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 3159-3168, Seattle, United States. Association for Computational Linguistics. · DOI
- Richard Blum and Christine Bresnahan. 2021. Linux Command Line and Shell Scripting Bible. Wiley.
- Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901.
- Federico Cassano, John Gouwar, Daniel Nguyen, Syd- ney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, Arjun Guha, Michael Greenberg, and Abhinav Jangda. 2023. MultiPL-E: A Scal- able and Polyglot Approach to Benchmarking Neural Code Generation . IEEE Transactions on Software Engineering, 49(07):3675-3691. · DOI · Paper page
- Oishik Chatterjee, Pooja Aggarwal, Suranjana Samanta, Ting Dai, Prateeti Mohapatra, Debanjana Kar, Ruchi Mahindru, Steve Barbieri, Eugen Postea, Brad Blancett, and Arthur De Magalhaes. 2024. Script- smith: A unified llm framework for enhancing it op- erations via automated bash script generation, assess- ment, and refinement. Preprint, arXiv:2409.17166. · arXiv
- Liguo Chen, Qi Guo, Hongrui Jia, Zhengran Zeng, Xin Wang, Yijiang Xu, Jian Wu, Yidong Wang, Qing Gao, Jindong Wang, Wei Ye, and Shikun Zhang. 2024. A survey on evaluating large language models in code generation tasks. Preprint, arXiv:2408.16498. · arXiv
- Heejae Chon, Seonghyeon Lee, Jinyoung Yeo, and Dongha Lee. 2024. Is functional correctness enough to evaluate code language models? exploring diver- sity of generated codes. Preprint, arXiv:2408.14504. · arXiv
- Berkeley Churchill, Oded Padon, Rahul Sharma, and Alex Aiken. 2019. Semantic program alignment for equivalence checking. In Proceedings of the 40th ACM SIGPLAN Conference on Programming Lan- guage Design and Implementation, PLDI 2019, pages 1027-1040, New York, NY, USA. Association for Computing Machinery. · DOI · Paper page
- Quchen Fu, Zhongwei Teng, Marco Georgaklis, Jules White, and Douglas Schmidt. 2023. Nl2cmd: An updated workflow for natural language to bash com- mands translation. Journal of Machine Learning Theory, Applications and Practice, 1. · DOI
- Quchen Fu, Zhongwei Teng, Jules White, and Dou- glas C. Schmidt. 2021. A transformer-based ap- proach for translating natural language to bash com- mands. In 2021 20th IEEE International Conference on Machine Learning and Applications (ICMLA), pages 1245-1248. · DOI
- Saibo Geng, Martin Josifoski, Maxime Peyrard, and Robert West. 2023. Grammar-constrained decoding for structured NLP tasks without finetuning. In Pro- ceedings of the 2023 Conference on Empirical Meth- ods in Natural Language Processing, pages 10932- 10952, Singapore. Association for Computational Linguistics. · DOI · Paper page
- David Gros. 2019. Ainix: An open platform for natural language interfaces to shell commands. Undergrad- uate Honors Thesis, Computer Science Department, University of Texas at Austin.
- Junjie Huang, Chenglong Wang, Jipeng Zhang, Cong Yan, Haotian Cui, Jeevana Priya Inala, Colin Clement, and Nan Duan. 2022. Execution-based evaluation for data science code generation models. In Proceedings of the Fourth Workshop on Data Science with Human- in-the-Loop (Language Advances), pages 28-36, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
- Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Day- iheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Kai Dang, et al. 2024. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186. · arXiv
- Sathvik Joel, Jie JW Wu, and Fatemeh H. Fard. 2024. A survey on llm-based code generation for low-resource and domain-specific programming lan- guages. Preprint, arXiv:2410.03981. · arXiv
- Xi Victoria Lin, Chenglong Wang, Luke Zettlemoyer, and Michael D. Ernst. 2018. NL2Bash: A corpus and semantic parser for natural language interface to the linux operating system. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
- Meta Llama. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783. · arXiv
- Nickil Maveli, Antonio Vergari, and Shay B. Co- hen. 2024. What can large language models cap- ture about code functional equivalence? Preprint, arXiv:2408.11081. · arXiv
- Dirk Merkel. 2014. Docker: lightweight linux con- tainers for consistent development and deployment. Linux journal, 2014(239):2. Microsoft. 2024. Microsoft ai shell. https: //learn.microsoft.com/en-us/powershell/ utility-modules/aishell/overview?view= ps-modules.
- Atharva Naik. 2024. On the limitations of embedding based methods for measuring functional correctness for code generation. Preprint, arXiv:2405.01580. · arXiv
- OpenAI. 2024. Gpt-4 technical report. Preprint, arXiv:2303.08774. · arXiv
- Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computa- tional Linguistics, ACL '02, pages 311-318, USA. Association for Computational Linguistics. · DOI · Paper page
- Jean E. Sammet. 1966. The use of english as a program- ming language. Commun. ACM, 9(3):228-230. · DOI
- Jie Shi, Sihang Jiang, Bo Xu, Jiaqing Liang, Yanghua Xiao, and Wei Wang. 2023. Shellgpt: Generative pre-trained transformer model for shell language un- derstanding. In 2023 IEEE 34th International Sym- posium on Software Reliability Engineering (ISSRE), pages 671-682. · DOI
- William E Shotts. 2019. The Linux Command Line: A Complete Introduction. No Starch Press.
- Yewei Song, Cedric Lothritz, Xunzhu Tang, Tegawendé Bissyandé, and Jacques Klein. 2024. Revisiting code similarity evaluation with abstract syntax tree edit distance. In Proceedings of the 62nd Annual Meet- ing of the Association for Computational Linguistics (Volume 2: Short Papers), pages 38-46, Bangkok, Thailand. Association for Computational Linguistics. · DOI
- Karen Sparck Jones. 1988. A statistical interpretation of term specificity and its application in retrieval, pages 132-142.
- Prashanth Vijayaraghavan, Luyao Shi, Stefano Am- brogio, Charles Mackin, Apoorva Nitsure, David Beymer, and Ehsan Degan. 2024. Vhdl-eval: A framework for evaluating large language models in vhdl code generation. Preprint, arXiv:2406.04379. · arXiv
- Ngoc Phuoc An Vo, Brent Paulovicks, and Vadim Sheinin. 2024. Execution-based evaluation of nat- ural language to bash and powershell for incident remediation. Preprint, arXiv:2405.06807. · arXiv
- John Yang, Akshara Prabhakar, Karthik Narasimhan, and Shunyu Yao. 2023. Intercode: standardizing and benchmarking interactive coding with execution feed- back. In Proceedings of the 37th International Con- ference on Neural Information Processing Systems, NIPS '23, Red Hook, NY, USA.
- Wenhao Zhu, Hongyi Liu, Qingxiu Dong, Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun Chen, and Lei Li. 2024b. Multilingual machine translation with large language models: Empirical results and anal- ysis. In Findings of the Association for Computa- tional Linguistics: NAACL 2024, pages 2765-2781, Mexico City, Mexico. Association for Computational Linguistics. · DOI
Cited by
Sorted by citations
Sabbagh, A. A., Abdo, E., Samrouth, K., Alkhatib, A. A., & Elkhodr, M. (2026) · The Journal of Supercomputing · doi:10.1007/s11227-026-08439-z · Cited by 2 · see citing works
From Voice to Shell: A SLM-Based Assistant for IoT Maintenance Tasks on the Edge
Flores, Á. G., Peño, C. B., Llumiguano, H., Ruiz, J. F.-B., Molina, F. J. V., & Calle, F. R. (2025) · IEEE Internet of Things Journal · doi:10.1109/jiot.2025.3632638
Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces
Merrill, M. A., Shaw, A. G., Carlini, N., Li, B., Raj, H., Bercovich, I., Shi, L., Shin, J. Y., Walshe, T., Buchanan, E. K., et al. (2026) · arXiv · doi:10.48550/arxiv.2601.11868
Yeşilyurt, M. E., & Tanrıöver, Ö. Ö. (2025) · International Conference on Problems of Cybernetics and Informatics (PCI) · doi:10.1109/pci66488.2025.11219789
Lightweight Statistical Model for Bash Command Recommendation
Tran, C., Hoang, D. N., & Vo, H. D. (2025) · RIVF International Conference on Computing and Communication Technologies (RIVF) · doi:10.1109/rivf68649.2025.11365121
Bridging the Gap Between Natural Language and CLI: An Intelligent Assistant Approach
Gupta, A. A., M, U. N., Kamath, L., Mahima, N. R., & Uma, D. (2025) · IEEE Region 10 Conference (TENCON) · doi:10.1109/tencon66050.2025.11375135
LLM Powered Command Line Interface Agent: A Comparative Analysis
Madhisha, S., Kabhinyasri, S., V, S, A. C. K., Pream, S., S., S. S., & D, A. C. (2026) · International Conference on Intelligent Systems for Communication, IoT and Security (ICISCoIS) · doi:10.1109/iciscois62701.2026.11447826
Kong, H., Ge, J., Liu, Z., Hu, D., Li, H., Li, L., & Wu, B. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11464572
- Authors
- Westenfelder, F., Hemberg, E., Moskal, S., O’Reilly, U.-M., & Chiricescu, S.
- Affiliations
- MIT Computer Science and Artificial Intelligence Laboratory (US)
- Draper Laboratory (US)
- Venue
- Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)
- Published
- 2025
- Pages
- 11135-11147
- OpenAlex
- W4411119257
- Cited by
- 7
- References
- 0
- Citations by year
- 2026: 3 · 2025: 4
- FWCI
- 6.67
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- large language models, shell scripting, code generation, programming language translation, natural language programming
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
