A comparison of code similarity analysers
Open accessThe DOI matches its Crossref record; 111 of the 111 references it cites were found in a registry or an index.
Abstract
Copying and pasting of source code is a common activity in software engineering. Often, the code is not copied as it is and it may be modified for various purposes; e.g. refactoring, bug fixing, or even software plagiarism. These code modifications could affect the performance of code similarity analysers including code clone and plagiarism detectors to some certain degree. We are interested in two types of code modification in this study: pervasive modifications, i.e. transformations that may have a global effect, and local modifications, i.e. code changes that are contained in a single method or code block. We evaluate 30 code similarity detection techniques and tools using five experimental scenarios for Java source code. These are (1) pervasively modified code, created with tools for source code and bytecode obfuscation, and boiler-plate code, (2) source code normalisation through compilation and decompilation using different decompilers, (3) reuse of optimal configurations over different data sets, (4) tool evaluation using ranked-based measures, and (5) local + global code modifications. Our experimental results show that in the presence of pervasive modifications, some of the general textual similarity measures can offer similar performance to specialised code similarity tools, whilst in the presence of boiler-plate code, highly specialised source code similarity detection techniques and tools outperform textual similarity measures. Our study strongly validates the use of compilation/decompilation as a normalisation technique. Its use reduced false classifications to zero for three of the tools. Moreover, we demonstrate that optimal configurations are very sensitive to a specific data set. After directly applying optimal configurations derived from one data set to another, the tools perform poorly on the new data set. The code similarity analysers are thoroughly evaluated not only based on several well-known pair-based and query-based error measures but also on each specific type of pervasive code modification. This broad, thorough study is the largest in existence and potentially an invaluable guide for future users of similarity detection in source code.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures















Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016
Individual Comparisons by Ranking Methods
Wilcoxon, F. (1945) · Biometrics Bulletin · doi:10.2307/3001968 · Cited by 15,354
Addelman, S. (1979) · Technometrics · doi:10.1080/00401706.1979.10489788 · Cited by 6,750
Fisher, R. A. (1936) · BMJ · doi:10.1136/bmj.1.3923.554-a · Cited by 4,980
An Introduction to Kolmogorov Complexity and Its Applications
Li, M., & Vitányi, P. (2019) · Texts in Computer Science · doi:10.1007/978-3-030-11298-1 · Cited by 3,681
Halstead, M. H. (1977) · CERN Document Server (European Organization for Nuclear Research) · Cited by 3,639
Halstead, M. H. (1977) · CERN Document Server (European Organization for Nuclear Research) · Cited by 2,339
CCFinder: a multilinguistic token-based code clone detection system for large scale source code
Kamiya, T., Kusumoto, S., & Inoue, K. (2002) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2002.1019480 · Cited by 1,711 · cited 6×
“ccfx (Kamiya et al. 2002) tokens and suffix tree matching deckard (Jiang et al. 2007b) characteristic vectors of AST optimised by LSH iclones (Göde and Koschke 2009) tokens and generalised suffix tree nicad (Roy and Cordy 2008) TXL and string comparison (LCS) simian (Harris 2015) line-based string…”
BackgroundMethodInfluentialAn Introduction to Kolmogorov Complexity and Its Applications
Li, M., & Vitányi, P. (2008) · Texts in Computer Science · doi:10.1007/978-0-387-49820-1 · Cited by 1,424
Clone detection using abstract syntax trees
Baxter, I. D., Yahin, A., Moura, L., Sant'Anna, M., & Bier, L. (2002) · Proceedings. International Conference on Software Maintenance (Cat. No. 98CB36272) · doi:10.1109/icsm.1998.738528 · Cited by 1,277 · cited 5×
“A few examples of well-known tree-based tools include CloneDR (Baxter et al. 1998), and Deckard (Jiang et al.”
BackgroundMethodInfluential
Research works printed but not linked: 2
- Manning CD, Raghavan P, Schutze H (2009) An introduction toinformation retrieval, vol 21. Cambridge University Press, Cambridge
- Nachenberg C (1996) Understanding and managing polymorphic viruses. The Symantec Enterprise Papers 30:16
Cited by
A Systematic Review on Code Clone Detection
Ain, Q. U., Butt, W. H., Anwar, M. W., Azam, F., & Maqbool, B. (2019) · IEEE Access · doi:10.1109/access.2019.2918202 · Cited by 165
“[76] compare the code similarity analyzers.”
BackgroundMethodInfluentialToxic Code Snippets on Stack Overflow
Ragkhitwetsagul, C., Krinke, J., Paixão, M. H. E., Bianco, G., & Oliveto, R. (2019) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2900307 · Cited by 136
“We are aware of effects of configurations to clone detection results and the importance of searching for optimised configurations in empirical clone studies [40], [41], [51], [52], [53].”
BackgroundSelf-citationComplement or Substitute? How AI Increases the Demand for Human SkillsPreprint
Mäkelä, E., & Stephany, F. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5153230 · Cited by 112
Nasrabadi, M. Z., Parsa, S. A., Ramezani, M., Roy, C. K., & Ekhtiarzadeh, M. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111796 · Cited by 129
Detecting LLM-Generated Text in Computing Education: Comparative Study for ChatGPT Cases
Orenstrakh, M. S., Karnalim, O., Suárez, C. A., & Liut, M. (2024) · 2024 IEEE 48th Annual Computers, Software, and Applications Conference (COMPSAC) · doi:10.1109/compsac61105.2024.00027 · Cited by 79
“In the context of plagiarism and collusion, automated detectors nullify common alterations that can be done without understanding the content [24, 43] and remove contents that are not evident for raising suspicion [48].”
Background
Metadata
- Authors
- Ragkhitwetsagul, C., Krinke, J., & Clark, D.
- Affiliations
- University College London (GB)
- Corresponding author
- Ragkhitwetsagul, Chaiyong
- Krinke, Jens
- Clark, David
- Venue
- Empirical Software Engineering
- Published
- 2017-10-25
- Volume
- 23
- Issue
- 4
- Pages
- 2464-2519
- ISSN
- 1382-3256 (ISSN-L)
- 1573-7616
- OpenAlex
- W2766078762
- Semantic Scholar
- ed1b5e97277bdb779ee015420f9e188a105f3df0
- Cited by
- 122
- References
- 110
- Citations by year
- 2026: 3 · 2025: 11 · 2024: 17 · 2023: 14 · 2022: 22 · 2021: 17 · 2020: 20 · 2019: 14 · 2018: 4
- FWCI
- 19.35
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Empirical Software Engineering (publishedVersion, cc-by)
- UCL Discovery (University College London) (submittedVersion)
- Funders
- Mahidol University
- Faculty of Information and Communication Technology, Mahidol University
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Social Sciences › Education › Academic integrity and plagiarism
- Physical Sciences › Computer Science › Hardware and Architecture › Physical Unclonable Functions (PUFs) and Hardware Security
- Keywords
- code clone detection, software plagiarism detection, code modification, obfuscation, boilerplate code, Java source code, false positive reduction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: A comparison of code similarity analysers. https://app.cochat.ai/papers/10.1007/s10664-017-9564-7
- The paper
- Ragkhitwetsagul, C., Krinke, J., & Clark, D. (2018). A comparison of code similarity analysers. Empirical Software Engineering. https://doi.org/10.1007/s10664-017-9564-7
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.