CoChat

Deep just-in-time defect prediction: how far are we?

Zeng, Zhengran, Zhang, Yuqun, Zhang, Haotian, Zhang, Lingming · 2021 · Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis
Cited by 102

The DOI matches its Crossref record; 4 of the 4 references it cites were found in a registry or an index.

Abstract

Defect prediction aims to automatically identify potential defective code with minimal human intervention and has been widely studied in the literature. Just-in-Time (JIT) defect prediction focuses on program changes rather than whole programs, and has been widely adopted in continuous testing. CC2Vec, state-of-the-art JIT defect prediction tool, first constructs a hierarchical attention network (HAN) to learn distributed vector representations of both code additions and deletions, and then concatenates them with two other embedding vectors representing commit messages and overall code changes extracted by the existing DeepJIT approach to train a model for predicting whether a given commit is defective. Although CC2Vec has been shown to be the state of the art for JIT defect prediction, it was only evaluated on a limited dataset and not compared with all representative baselines. Therefore, to further investigate the efficacy and limitations of CC2Vec, this paper performs an extensive study of CC2Vec on a large-scale dataset with over 310,370 changes (8.3 X larger than the original CC2Vec dataset). More specifically, we also empirically compare CC2Vec against DeepJIT and representative traditional JIT defect prediction techniques. The experimental results show that CC2Vec cannot consistently outperform DeepJIT, and neither of them can consistently outperform traditional JIT defect prediction. We also investigate the impact of individual traditional defect prediction features and find that the added-line-number feature outperforms other traditional features. Inspired by this finding, we construct a simplistic JIT defect prediction approach which simply adopts the added-line-number feature with the logistic regression classifier. Surprisingly, such a simplistic approach can outperform CC2Vec and DeepJIT in defect prediction, and can be 81k X/120k X faster in training/testing. Furthermore, the paper also provides various practical guidelines for advancing JIT defect prediction in the near future.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3460319.3464819

Cites

4 of 4 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 4 references

Cited by

  • An extensive study on pre-trained models for program understanding and generation

    Zeng, Z., Tan, H., Zhang, H., Li, J., Zhang, Y., & Zhang, L. (2022) · Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3533767.3534390 · Cited by 153

    “In recent years, researchers have explored various techniques in the intersection of software engineering and machine learning [5, 8, 23, 36, 41, 70, 72, 78, 81, 82].”
    BackgroundSelf-citation
  • Free lunch for testing: fuzzing deep-learning libraries from open source

    Wei, A., Deng, Y., Yang, C., & Zhang, L. (2022) · Proceedings of the 44th International Conference on Software Engineering · doi:10.1145/3510003.3510041 · Cited by 106

    “Deep Learning (DL) has been playing a significant role in various application domains, including image classification [39, 62], natural language processing [33, 35], game playing [61], and software engineering [23, 45, 74, 75].”
    BackgroundSelf-citation
  • Fuzzing deep-learning libraries via automated relational API inference

    Deng, Y., Yang, C., Wei, A., & Zhang, L. (2022) · Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3540250.3549085 · Cited by 86

    “Recent years have witnessed the surge of deep learning (DL) in a variety of applications, including computer vision [32, 60], natural language processing [24, 28], robotics [23, 35], bioinformatics [46, 57], and software engineering [25, 29, 36, 37, 45, 68, 78, 79, 81, 82].”
    BackgroundSelf-citation

See all 102

Metadata

Authors
Zeng, Z., Zhang, Y., Zhang, H., & Zhang, L.
Venue
Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis
Publisher
ACM
Published
2021-07-11
Pages
427-438
OpenAlex
W3178061567
Cited by
102
References
4
FWCI
14.43
Type
conference paper
Language
en
Open access
closed

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Deep just-in-time defect prediction: how far are we? https://app.cochat.ai/papers/10.1145/3460319.3464819
The paper
Zeng, Z., Zhang, Y., Zhang, H., & Zhang, L. (2021). Deep just-in-time defect prediction: how far are we? Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis. https://doi.org/10.1145/3460319.3464819

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.