CoChat

Learning to Code with Context: A Study-Based Approach

Open access

The DOI matches its Crossref record; 23 of the 53 references it cites were found in a registry or an index, and 1 are not checked yet; none of the 10 sources its claims cite is checked for support yet.

Abstract

The rapid emergence of generative AI tools is transforming software development. Consequently, software engineering education must adapt to ensure that students not only learn traditional development methods but also understand how to use these new technologies effectively and responsibly. In particular, project-based courses provide an effective setting in which to explore and evaluate the integration of AI assistance into real-world development practices. This paper presents our approach and a user study conducted in the context of a university programming project in which students collaboratively developed computer games. The study investigates how participants used generative AI tools across different phases of the software development process, identifies the tasks for which these tools were perceived as most useful, and analyzes the challenges students encountered. Building on these insights, we further examine a repository-aware, locally deployed large language model (LLM) assistant designed to provide project-contextualized support. The system employs retrieval-augmented generation (RAG) to ground its responses in relevant documentation and source code, thereby enabling a qualitative analysis of model behavior, parameter sensitivity, and common failure modes. These findings deepen our understanding of context-aware AI support in educational software projects and inform the future integration of AI-based assistance into software engineering curricula.

Checked 2026-10-11 · 1 reference not checked yet

Overview

CoChat Summary

## TL;DR

This paper studies AI use in a university programming project and finds that students use generative AI most in coding/testing, but text-heavy tasks benefit more than code comprehension or diagram work. It then shows that a locally deployed, repository-aware RAG assistant can produce more usable, repository-aligned outputs than generic assistants.

## Key findings

  • Among 38 participants, 34 used AI tools during the project, an adoption rate of 89.4%; 4 did not use AI, citing lack of need or personal reservations, and 1 also said they were unaware of the available tools.
  • AI use varied by phase: Preparation 22/38, Analysis 13/38, Design 10/38, and Coding & Testing 33/38; ChatGPT was used by all students who used AI tools, and GitHub Copilot was the next most common tool, especially in coding.
  • In the Preparation phase, comprehension was the most common AI task (n=9), but 6 of those 9 reported “inaccurate results” and 2 reported “missing context.”
  • In the repository-aware assistant evaluation, 42 runs were conducted across 15 sampling configurations and 6 model variants; the repository-aware setup “consistently produced usable, repository-aligned outputs,” and in most runs artifacts were integrated with limited follow-up corrections.
  • For sampling, Hallucination was the most frequent defect, occurring in 14 of 15 runs; the paper says the most stable configuration was T ≈ 0.5, top p = 0, yielding the fewest defects.
  • In the model sweep, every model except Mistral-7B showed at least one marked Hallucination instance; Mistral-7B was described as unsuitable because the run failed to produce complete solutions for both tasks.

## Methods

  • User study: structured questionnaire administered during the final team presentation of a university programming project; 38 participants out of 49 enrolled students participated, and the study asked about AI use across Preparation, Analysis, Design, and Coding & Testing phases plus overall usefulness and future intentions.
  • Case study: repository-aware, locally deployed LLM assistant grounded in project documentation and source code via Retrieval-Augmented Generation (RAG), running on a dedicated on-campus NVIDIA DGX-H100 system with Docker containers.
  • Evaluation: 42 single-run model/task configurations, with 15 sampling configurations and 6 model variants evaluated on 2 representative preparation tasks; outputs were analyzed against a predefined defect catalog.

## Limitations

  • The paper states that the case-study evaluation used a single-run protocol per configuration and task, which constrains reproducibility and the stability of observed effects.
  • The analysis relies on manual coding against a predefined defect catalog, and the paper says the lack of a standardized benchmark introduces subjectivity and potential coder bias.
  • External validity is constrained by only two tasks from a single project, and the paper notes that other architectures or task types may yield different patterns.
  • Evident from the text: the user study is self-reported questionnaire data from one course cohort, so the findings depend on participants’ recollections and perceptions rather than direct observation.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.3390/software5020027

Cites

23 of 53 references verified0 citation errors · 29 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

0 Valid · 0 Not valid · 10 Not checked · 198 Own results and interpretations

98 more claims

Create a free account to see all 208 claims and to verify the 10 references still open.

See all claims — free account
  • It may state two separate claims as one. 64
  • Probably not a substantial claim. 71

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

1 of 1

Citations

Cites

Show all 53 references

Research works printed but not linked: 1

  1. Borghoff, U.M., Minas, M., and Schopp, J. (2025). Learning to Code with Context: A Study-Based Approach. arXiv.

Metadata

Authors
Borghoff, U. M., Minas, M., & Schopp, J.
Affiliations
Universität der Bundeswehr München (DE)
Corresponding author
Borghoff, Uwe M.
Venue
Software
Publisher
MDPI AG
Published
2026-06-21
Volume
5
Issue
2
Pages
27
ISSN
2674-113X (ISSN-L)
OpenAlex
W4417142121
Cited by
0
References
52
FWCI
0.00
Type
journal article
Language
en
Open access
Locations
  • Software (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Social Sciences › Education › Artificial Intelligence in Education
Keywords
generative AI in software development, large language models, retrieval-augmented generation, AI-assisted programming, software engineering education, project-based learning, game development, student perceptions of AI tools, AI failure modes, parameter sensitivity

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Learning to Code with Context: A Study-Based Approach. https://app.cochat.ai/papers/10.3390/software5020027
The paper
Borghoff, U. M., Minas, M., & Schopp, J. (2026). Learning to Code with Context: A Study-Based Approach. Software. https://doi.org/10.3390/software5020027

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.