CoChat
Sign Up
Cited by 8Software Engineering ResearchSoftware Engineering Techniques and PracticesTopic Modeling

Can Developers Prompt? A Controlled Experiment for Code Documentation Generation

Open access
Kruse, Hans-Alexander, Puhlfürß, Tim, Maalej, Walid·2024·2024 IEEE International Conference on Software Maintenance and Evolution (ICSME)
DOI 10.1109/icsme58944.2024.00058
Open this paper in CoChat — free accountOpen PDF

Abstract

Large language models (LLMs) bear great potential for automating tedious development tasks such as creating and maintaining code documentation. However, it is unclear to what extent developers can effectively prompt LLMs to create concise and useful documentation. We report on a controlled experiment with 20 professionals and 30 computer science students tasked with code documentation generation for two Python functions. The experimental group freely entered ad-hoc prompts in a Chat- GPT-like extension of Visual Studio Code, while the control group executed a predefined few-shot prompt. Our results reveal that professionals and students were unaware of or unable to apply prompt engineering techniques. Especially students perceived the documentation produced from ad-hoc prompts as significantly less readable, less concise, and less helpful than documentation from prepared prompts. Some professionals produced higher quality documentation by just including the keyword Docstring in their ad-hoc prompts. While students desired more support in formulating prompts, professionals appreciated the flexibility of ad-hoc prompting. Participants in both groups rarely assessed the output as perfect. Instead, they understood the tools as support to iteratively refine the documentation. Further research is needed to understand which prompting skills and preferences developers have and which support they need for certain tasks.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/icsme58944.2024.00058

Cites

8 of 8 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • This supports prior findings that some students do not understand what information precisely needs to be shared with the chatbot.

    “This supports prior findings by Mailach et al. [2025] and Kruse et al. [2024] that some students do not understand what information precisely needs to be shared with the chatbot.”

    From How Do Programming Students Use Generative AI? (2025) · cites this paper as Kruse et al. [2024]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 45

  1. E. Aghajani, C. Nagy, O. L. Vega-Márquez, M. Linares-Vásquez, L. Moreno, G. Bavota, and M. Lanza, "Software documentation issues unveiled," in 2019 IEEE/ACM 41st International Conference on Software Engineering. New York, NY, USA: IEEE, 2019, pp. 1199-1210.
  2. T. Roehm, R. Tiarks, R. Koschke, and W. Maalej, "How do professional developers comprehend software?" in 2012 34th International Confer- ence on Software Engineering. New York, NY, USA: IEEE, 2012, pp. 255-265.
  3. I. Steinmacher, C. Treude, and M. A. Gerosa, "Let me in: Guidelines for the successful onboarding of newcomers to open source projects," IEEE Software, vol. 36, no. 4, pp. 41-49, 2019.
  4. C. Stanik, L. Montgomery, D. Martens, D. Fucci, and W. Maalej, "A simple NLP-based approach to support onboarding and retention in open source communities," in 2018 IEEE International Conference on Software Maintenance and Evolution, 2018, pp. 172-182.
  5. F. Zampetti, G. Fucci, A. Serebrenik, and M. Di Penta, "Self-admitted technical debt practices: A comparison between industry and open- source," Empirical Software Engineering, vol. 26, no. 6, pp. 1-32, 2021.
  6. E. Aghajani, C. Nagy, M. Linares-Vásquez, L. Moreno, G. Bavota, M. Lanza, and D. C. Shepherd, "Software documentation: The prac- titioners' perspective," in Proceedings of the ACM/IEEE 42nd Interna- tional Conference on Software Engineering. New York, NY, USA: Association for Computing Machinery, 2020, p. 590-601.
  7. S. Rai, R. C. Belwal, and A. Gupta, "A review on source code docu- mentation," ACM Transactions on Intelligent Systems and Technology, vol. 13, no. 5, Jun. 2022.
  8. C. Ebert and P. Louridas, "Generative AI for software practitioners," IEEE Software, vol. 40, no. 4, pp. 30-38, 2023.
  9. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert- Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei, "Language models are few-shot learners," in Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, Eds., vol. 33. Red Hook, NY, USA: Curran Associates, Inc., 2020, pp. 1877-1901.
  10. R. "Logan IV, I. Balazevic, E. Wallace, F. Petroni, S. Singh, and S. Riedel, ""cutting down on prompts and parameters: Simple few-shot learning with language models"," in "Findings of the Association for Computational Linguistics: ACL 2022", S. "Muresan, P. Nakov, and A. Villavicencio, Eds. "Dublin, Ireland": "Association for Computa- tional Linguistics", May "2022", pp. "2824-2835".
  11. T. Ahmed and P. Devanbu, "Few-shot training LLMs for project- specific code-summarization," in Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering. New York, NY, USA: Association for Computing Machinery, 2023.
  12. C. Wohlin, P. Runeson, M. Höst, M. C. Ohlsson, B. Regnell, and A. Wesslén, Experimentation in software engineering. Berlin, Hei- delberg: Springer, 2012.
  13. O. Alhadreti and P. Mayhew, "Rethinking thinking aloud: A comparison of three think-aloud protocols," in Proceedings of the 2018 CHI Confer- ence on Human Factors in Computing Systems. New York, NY, USA: Association for Computing Machinery, 2018, p. 1-12.
  14. B. Laugwitz, T. Held, and M. Schrepp, "Construction and evaluation of a user experience questionnaire," in HCI and Usability for Education and Work, A. Holzinger, Ed. Berlin, Heidelberg: Springer Berlin Heidelberg, 2008, pp. 63-76.
  15. R. Macefield, "Usability studies and the Hawthorne effect," J. Usability Studies, vol. 2, no. 3, p. 145-154, May 2007.
  16. P. Rani, A. Blasi, N. Stulova, S. Panichella, A. Gorla, and O. Nierstrasz, "A decade of code comment quality assessment: A systematic literature review," Journal of Systems and Software, vol. 195, p. 111515, 2023.
  17. X. Hou, Y. Zhao, Y. Liu, Z. Yang, K. Wang, L. Li, X. Luo, D. Lo, J. Grundy, and H. Wang, "Large language models for software engi- neering: A systematic literature review," 2023.
  18. B. Derrick, D. Toher, and P. White, "Why Welch's test is Type I error robust," The Quantitative Methods for Psychology, vol. 12, no. 1, pp. 30-38, 2016.
  19. N. Nachar, "The mann-whitney u: A test for assessing whether two independent samples come from the same distribution," Tutorials in Quantitative Methods for Psychology, vol. 4, no. 1, pp. 13-20, 2008.
  20. N. Mohd Razali and B. Yap, "Power comparisons of Shapiro-Wilk, Kolmogorov-Smirnov, Lilliefors and Anderson-Darling tests," Journal of Statistical Modeling and Analytics, vol. 2, no. 1, pp. 21-33, 2011.
  21. M. A. Cascio, E. Lee, N. Vaudrin, and D. A. Freedman, "A team- based approach to open coding: Considerations for creating intercoder consensus," Field Methods, vol. 31, no. 2, pp. 116-130, 2019.
  22. M. Torricelli, M. Martino, A. Baronchelli, and L. M. Aiello, "The role of interface design on prompt-mediated creativity in generative ai," in Proceedings of the 16th ACM Web Science Conference. New York, NY, USA: Association for Computing Machinery, 2024, p. 235-240.
  23. W. Maalej and M. P. Robillard, "Patterns of knowledge in API reference documentation," IEEE Transactions on Software Engineering, vol. 39, no. 9, pp. 1264-1282, 2013.
  24. W. "Maalej and H.-J. Happel, ""a lightweight approach for knowl- edge sharing in distributed software teams"," in "Practical Aspects of Knowledge Management", T. "Yamaguchi, Ed. "Berlin, Heidelberg": "Springer Berlin Heidelberg", "2008", pp. "14-25".
  25. W. Maalej, "From RSSE to BotSE: Potentials and challenges revisited after 15 years," in 2023 IEEE/ACM 5th International Workshop on Bots in Software Engineering, 2023, pp. 19-22.
  26. H. Tang and S. Nadi, "Evaluating software documentation quality," in 2023 IEEE/ACM 20th International Conference on Mining Software Repositories, 2023, pp. 67-78.
  27. W. Maalej, R. Tiarks, T. Roehm, and R. Koschke, "On the comprehen- sion of program comprehension," ACM Trans. Softw. Eng. Methodol., vol. 23, no. 4, sep 2014.
  28. D. Fucci, A. Mollaalizadehbahnemiri, and W. Maalej, "On using ma- chine learning to identify knowledge in API reference documentation," in Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. New York, NY, USA: Association for Computing Machinery, 2019, p. 109-119.
  29. Z. "Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang, and M. Zhou, ""CodeBERT: A pre-trained model for programming and natural languages"," in "Findings of the Association for Computational Linguistics: EMNLP 2020", T. "Cohn, Y. He, and Y. Liu, Eds. "Online": "Association for Computational Linguistics", "Nov." "2020", pp. "1536-1547".
  30. M. Geng, S. Wang, D. Dong, H. Wang, G. Li, Z. Jin, X. Mao, and X. Liao, "Large language models are few-shot summarizers: Multi- intent comment generation via in-context learning," in Proceedings of the IEEE/ACM 46th International Conference on Software Engineering. New York, NY, USA: Association for Computing Machinery, 2024.
  31. J. Wei, X. Wang, D. Schuurmans, M. Bosma, b. ichter, F. Xia, E. Chi, Q. V. Le, and D. Zhou, "Chain-of-thought prompting elicits reasoning in large language models," in Advances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, Eds., vol. 35. Curran Associates, Inc., 2022, pp. 24 824-24 837.
  32. B. "Meskó, ""prompt engineering as an important emerging skill for medical professionals: Tutorial"," "Journal of Medical Internet Re- search", vol. "25", p. "e50638", "Oct." "2023".
  33. T. F. Heston and C. Khun, "Prompt engineering in medical education," International Medical Education, vol. 2, no. 3, pp. 198-205, 2023.
  34. G. Sridhara, E. Hill, D. Muppaneni, L. Pollock, and K. Vijay-Shanker, "Towards automatically generating summary comments for Java meth- ods," in Proceedings of the 25th IEEE/ACM International Conference on Automated Software Engineering. New York, NY, USA: Association for Computing Machinery, 2010, p. 43-52.
  35. P. W. McBurney and C. McMillan, "Automatic documentation genera- tion via source code summarization of method context," in Proceedings of the 22nd International Conference on Program Comprehension. New York, NY, USA: Association for Computing Machinery, 2014, p. 279-290.
  36. X. Hu, G. Li, X. Xia, D. Lo, and Z. Jin, "Deep code comment generation," in Proceedings of the 26th Conference on Program Compre- hension. New York, NY, USA: Association for Computing Machinery, 2018, p. 200-210.
  37. U. Alon, O. Levy, and E. Yahav, "code2seq: Generating sequences from structured representations of code," in International Conference on Learning Representations. Online: OpenReview, 2019.
  38. A. LeClair, S. Haque, L. Wu, and C. McMillan, "Improved code summarization via a graph neural network," in Proceedings of the 28th International Conference on Program Comprehension. New York, NY, USA: Association for Computing Machinery, 2020, p. 184-195.
  39. S. Gao, C. Gao, Y. He, J. Zeng, L. Nie, X. Xia, and M. Lyu, "Code structure-guided transformer for source code summarization," ACM Trans. Softw. Eng. Methodol., vol. 32, no. 1, Feb. 2023.
  40. Y. Wan, Z. Zhao, M. Yang, G. Xu, H. Ying, J. Wu, and P. S. Yu, "Improving automatic source code summarization via deep reinforce- ment learning," in Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering. New York, NY, USA: Association for Computing Machinery, 2018, p. 397-407.
  41. R. Durelli, V. Durelli, R. Bettio, D. Dias, and A. Goldman, "Divinator: A visual studio code extension to source code summarization," in Anais do X Workshop de Visualizac ¸ão, Evoluc ¸ão e Manutenc ¸ão de Software. Porto Alegre, RS, Brasil: SBC, 2022, pp. 1-5.
  42. T. Puhlfürß, L. Montgomery, and W. Maalej, "An exploratory study of documentation strategies for product features in popular GitHub projects," in 2022 IEEE International Conference on Software Main- tenance and Evolution, 2022, pp. 379-383.
  43. S. Aljumah and L. Berriche, "Bi-LSTM-based neural source code summarization," Applied Sciences, vol. 12, no. 24, 2022.
  44. Y. Gao and C. Lyu, "M2TS: Multi-scale multi-modal approach based on transformer for source code summarization," in Proceedings of the 30th IEEE/ACM International Conference on Program Comprehension. New York, NY, USA: Association for Computing Machinery, 2022, p. 24-35.
  45. H. Guo, X. Chen, Y. Huang, Y. Wang, X. Ding, Z. Zheng, X. Zhou, and H.-N. Dai, "Snippet comment generation based on code context expansion," ACM Trans. Softw. Eng. Methodol., vol. 33, no. 1, Nov. 2023.

Cited by

Sorted by citations

Authors
Kruse, H.-A., Puhlfürß, T., & Maalej, W.
Affiliations
Universität Hamburg (DE)
ORCID
Venue
2024 IEEE International Conference on Software Maintenance and Evolution (ICSME)
Publisher
IEEE
Published
2024-10-06
Pages
574-586
ISSN
1063-6773 (ISSN-L)
DOI
10.1109/icsme58944.2024.00058
OpenAlex
W4405602033
Cited by
8
References
8
Citations by year
2026: 3 · 2025: 5
FWCI
3.04
Type
conference paper
Language
en
Open access
Licence
Locations
  • IEEE International Conference on Software Maintenance (ICSM) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
prompt engineering, large language models, few-shot prompting, developer skills, Python functions, Visual Studio Code extension

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review