CoChat

A comparison of code similarity analysers

Open access
Ragkhitwetsagul, Chaiyong, Krinke, Jens, Clark, David · 2018 · Empirical Software Engineering

The DOI matches its Crossref record; 111 of the 111 references it cites were found in a registry or an index.

Abstract

Copying and pasting of source code is a common activity in software engineering. Often, the code is not copied as it is and it may be modified for various purposes; e.g. refactoring, bug fixing, or even software plagiarism. These code modifications could affect the performance of code similarity analysers including code clone and plagiarism detectors to some certain degree. We are interested in two types of code modification in this study: pervasive modifications, i.e. transformations that may have a global effect, and local modifications, i.e. code changes that are contained in a single method or code block. We evaluate 30 code similarity detection techniques and tools using five experimental scenarios for Java source code. These are (1) pervasively modified code, created with tools for source code and bytecode obfuscation, and boiler-plate code, (2) source code normalisation through compilation and decompilation using different decompilers, (3) reuse of optimal configurations over different data sets, (4) tool evaluation using ranked-based measures, and (5) local + global code modifications. Our experimental results show that in the presence of pervasive modifications, some of the general textual similarity measures can offer similar performance to specialised code similarity tools, whilst in the presence of boiler-plate code, highly specialised source code similarity detection techniques and tools outperform textual similarity measures. Our study strongly validates the use of compilation/decompilation as a normalisation technique. Its use reduced false classifications to zero for three of the tools. Moreover, we demonstrate that optimal configurations are very sensitive to a specific data set. After directly applying optimal configurations derived from one data set to another, the tools perform poorly on the new data set. The code similarity analysers are thoroughly evaluated not only based on several well-known pair-based and query-based error measures but also on each specific type of pervasive code modification. This broad, thorough study is the largest in existence and potentially an invaluable guide for future users of similarity detection in source code.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1007/s10664-017-9564-7

Cites

111 of 111 references verified0 citation errors · 0 not checkable

Statistics

Figures

Image 1
CC-BY · From the PDF
Table 1 List of pervasive code modifications offered by our source code and bytecode obfuscator, and compiler/decompilers
Table 1 List of pervasive code modifications offered by our source code and bytecode obfuscator, and compiler/decompilersCC-BY · From the PDF
Fig. 5 Test data generation process
Fig. 5 Test data generation processCC-BY · From the PDF
Image 4
CC-BY · From the PDF
Image 5
CC-BY · From the PDF
Fig. 8 The (zoomed) ROC curves of the 10 tools that have the highest area under the curve (AUC) for SOCO
Fig. 8 The (zoomed) ROC curves of the 10 tools that have the highest area under the curve (AUC) for SOCOCC-BY · From the PDF
Image 7
CC-BY · From the PDF
Image 8
CC-BY · From the PDF
Fig. 11 Comparison of tool performances (F1-score) before and after decompilation
Fig. 11 Comparison of tool performances (F1-score) before and after decompilationCC-BY · From the PDF
Fig. 12 Precision-at-n of the tools according to varied numbers of n against generated data set
Fig. 12 Precision-at-n of the tools according to varied numbers of n against generated data setCC-BY · From the PDF
Image 11
CC-BY · From the PDF
Image 12
CC-BY · From the PDF
Image 13
CC-BY · From the PDF
Image 14
CC-BY · From the PDF
Image 15
CC-BY · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 111 references

Research works printed but not linked: 2

  1. Manning CD, Raghavan P, Schutze H (2009) An introduction toinformation retrieval, vol 21. Cambridge University Press, Cambridge
  2. Nachenberg C (1996) Understanding and managing polymorphic viruses. The Symantec Enterprise Papers 30:16

Cited by

See all 134

Metadata

Authors
Ragkhitwetsagul, C., Krinke, J., & Clark, D.
Affiliations
University College London (GB)
Corresponding author
  • Ragkhitwetsagul, Chaiyong
  • Krinke, Jens
  • Clark, David
Venue
Empirical Software Engineering
Published
2017-10-25
Volume
23
Issue
4
Pages
2464-2519
ISSN
  • 1382-3256 (ISSN-L)
  • 1573-7616
OpenAlex
W2766078762
Cited by
122
References
110
Citations by year
2026: 3 · 2025: 11 · 2024: 17 · 2023: 14 · 2022: 22 · 2021: 17 · 2020: 20 · 2019: 14 · 2018: 4
FWCI
19.35
Type
journal article
Language
en
Open access
Locations
  • Empirical Software Engineering (publishedVersion, cc-by)
  • UCL Discovery (University College London) (submittedVersion)
Funders
  • Mahidol University
  • Faculty of Information and Communication Technology, Mahidol University
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Social Sciences › Education › Academic integrity and plagiarism
  • Physical Sciences › Computer Science › Hardware and Architecture › Physical Unclonable Functions (PUFs) and Hardware Security
Keywords
code clone detection, software plagiarism detection, code modification, obfuscation, boilerplate code, Java source code, false positive reduction

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: A comparison of code similarity analysers. https://app.cochat.ai/papers/10.1007/s10664-017-9564-7
The paper
Ragkhitwetsagul, C., Krinke, J., & Clark, D. (2018). A comparison of code similarity analysers. Empirical Software Engineering. https://doi.org/10.1007/s10664-017-9564-7

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.