Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Cited by 12

The DOI matches its Crossref record; 17 of the 17 references it cites were found in a registry or an index.

Lessons from Building StackSpot AI: A Contextualized AI Coding Assistant

Pinto, Gustavo, De Souza, Cleidson, Neto, Joao Batista, Souza, Alberto, Gotto, Tarci­sio, Monteiro, Edward · 2024 · Proceedings of the 46th International Conference on Software Engineering: Software Engineering in Practice

Abstract

With their exceptional natural language processing capabilities, tools based on Large Language Models (LLMs) like ChatGPT and CoPilot have swiftly become indispensable resources in the software developer's toolkit. While recent studies suggest the potential productivity gains these tools can unlock, users still encounter drawbacks, such as generic or incorrect answers. Additionally, the pursuit of improved responses often leads to extensive prompt engineering efforts, diverting valuable time from writing code that delivers actual value. To address these challenges, a new breed of tools, built atop LLMs, is emerging. These tools aim to mitigate drawbacks by employing techniques like fine-tuning or enriching user prompts with contextualized information.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3639477.3639751

Cites

17 of 17 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 17 references

Research works printed but not linked: 20

  1. Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Collisson, Jina Suh, Shamsi Iqbal, Paul Bennett, Kori Inkpen, Jaime Teevan, Ruth Kikin-Gil, and Eric Horvitz. 2019. Guidelines for Human-AI Interaction. In CHI 2019. ACM. https://www.microsoft.com/en-us/research/publication/guidelines-for-human-ai-interaction/ CHI 2019 Honorable Mention Award.
  2. Samuel R Bowman. 2023. Eight things to know about large language models. arXiv preprint arXiv:2304.00612 (2023). · arXiv
  3. Yupeng Chang Xu Wang Jindong Wang Yuan Wu Kaijie Zhu Hao Chen Linyi Yang Xiaoyuan Yi Cunxiang Wang Yidong Wang et al. 2023. A survey on evaluation of large language models. arXiv preprint arXiv:2307.03109 (2023). · arXiv
  4. Victoria Clarke and Virginia Braun. 2013. Successful Qualitative Research: A Practical Guide for Beginners. Sage, London.
  5. Jean-Baptiste Döderlein, Mathieu Acher, Djamel Eddine Khelladi, and Benoit Combemale. 2023. Piloting Copilot and Codex: Hot Temperature, Cold Prompts, or Black Magic? arXiv:2210.14699 [cs.SE] · arXiv
  6. Shahul Es, Jithin James, Luis Espinosa-Anke, and Steven Schockaert. 2023. Ra-gas: Automated evaluation of retrieval augmented generation. arXiv preprint arXiv:2309.15217 (2023). · arXiv
  7. 10.48550/arXiv.2302.05319 · arXiv · doi:10.48550/arxiv.2302.05319
  8. 10.48550/arXiv.2308.10620 · DOI · Paper page
  9. CS Krishna. 2023. Prompt Generate Train (PGT): A framework for few-shot domain adaptation alignment and uncertainty calibration of a retriever augmented generation (RAG) model for domain specific open book question-answering. arXiv preprint arXiv:2307.05915 (2023). · arXiv
  10. Patrick S. H. Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (Eds.). https://proceedings.neurips.cc/paper/2020/hash/6b493230205f780e1bc26945df7481e5-Abstract.html
  11. Jia Li Yunfei Zhao Yongmin Li Ge Li and Zhi Jin. 2023. AceCoder: Utilizing Existing Code to Enhance Code Generation. arXiv:2303.17780 [cs.SE] · arXiv
  12. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out. 74--81.
  13. 10.48550/arXiv.2308.02828 · DOI · Paper page
  14. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics. 311--318.
  15. 10.1145/3581641.3584037 · DOI · Paper page
  16. Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. 2023. Large language models in medicine. Nature medicine (2023), 1--11.
  17. 10.5555/3295222.3295349 · doi:10.5555/3295222.3295349
  18. 10.1145/3397481.3450656 · DOI · Paper page
  19. Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David Rosenberg, and Gideon Mann. 2023. Bloomberggpt: A large language model for finance. arXiv preprint arXiv:2303.17564 (2023). · arXiv
  20. 10.48550/arXiv.2308.09932 · DOI

Cited by

See all 12

Metadata

Authors
Pinto, G., De Souza, C., Neto, J. B., Souza, A., Gotto, T., & Monteiro, E.
Affiliations
  • Federal University of Pará, Belém, Pará, Brazil
  • Federal University of Para, Belém, Pará, Brazil
  • Zup Innovation, Uberlandia, Brazil
  • StackSpot, Sao Paulo, Brazil
Venue
Proceedings of the 46th International Conference on Software Engineering: Software Engineering in Practice
Publisher
ACM
Published
2024-04-14
Pages
408-417
Cited by
12
References
17
Funders
  • CNPq (420406/2023-9, 442779/2023-2, 465614/2014-0, 308623/2022-3)
  • FAPESPA (053/2021)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Lessons from Building StackSpot AI: A Contextualized AI Coding Assistant. https://app.cochat.ai/papers/10.1145/3639477.3639751
The paper
Pinto, G., De Souza, C., Neto, J. B., Souza, A., Gotto, T., & Monteiro, E. (2024). Lessons from Building StackSpot AI: A Contextualized AI Coding Assistant. Proceedings of the 46th International Conference on Software Engineering: Software Engineering in Practice. https://doi.org/10.1145/3639477.3639751

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.