A comparison of code similarity analysers
Open accessThe DOI matches its Crossref record; 111 of the 111 references it cites were found in a registry or an index.
Abstract
Copying and pasting of source code is a common activity in software engineering. Often, the code is not copied as it is and it may be modified for various purposes; e.g. refactoring, bug fixing, or even software plagiarism. These code modifications could affect the performance of code similarity analysers including code clone and plagiarism detectors to some certain degree. We are interested in two types of code modification in this study: pervasive modifications, i.e. transformations that may have a global effect, and local modifications, i.e. code changes that are contained in a single method or code block. We evaluate 30 code similarity detection techniques and tools using five experimental scenarios for Java source code. These are (1) pervasively modified code, created with tools for source code and bytecode obfuscation, and boiler-plate code, (2) source code normalisation through compilation and decompilation using different decompilers, (3) reuse of optimal configurations over different data sets, (4) tool evaluation using ranked-based measures, and (5) local + global code modifications. Our experimental results show that in the presence of pervasive modifications, some of the general textual similarity measures can offer similar performance to specialised code similarity tools, whilst in the presence of boiler-plate code, highly specialised source code similarity detection techniques and tools outperform textual similarity measures. Our study strongly validates the use of compilation/decompilation as a normalisation technique. Its use reduced false classifications to zero for three of the tools. Moreover, we demonstrate that optimal configurations are very sensitive to a specific data set. After directly applying optimal configurations derived from one data set to another, the tools perform poorly on the new data set. The code similarity analysers are thoroughly evaluated not only based on several well-known pair-based and query-based error measures but also on each specific type of pervasive code modification. This broad, thorough study is the largest in existence and potentially an invaluable guide for future users of similarity detection in source code.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures















Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016
Individual Comparisons by Ranking Methods
Wilcoxon, F. (1945) · Biometrics Bulletin · doi:10.2307/3001968 · Cited by 15,354
Addelman, S. (1979) · Technometrics · doi:10.1080/00401706.1979.10489788 · Cited by 6,750
Fisher, R. A. (1936) · BMJ · doi:10.1136/bmj.1.3923.554-a · Cited by 4,980
An Introduction to Kolmogorov Complexity and Its Applications
Li, M., & Vitányi, P. (2019) · Texts in Computer Science · doi:10.1007/978-3-030-11298-1 · Cited by 3,681
Halstead, M. H. (1977) · CERN Document Server (European Organization for Nuclear Research) · Cited by 3,638
Halstead, M. H. (1977) · CERN Document Server (European Organization for Nuclear Research) · Cited by 2,339
CCFinder: a multilinguistic token-based code clone detection system for large scale source code
Kamiya, T., Kusumoto, S., & Inoue, K. (2002) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2002.1019480 · Cited by 1,711 · cited 6×
“ccfx (Kamiya et al. 2002) tokens and suffix tree matching deckard (Jiang et al. 2007b) characteristic vectors of AST optimised by LSH iclones (Göde and Koschke 2009) tokens and generalised suffix tree nicad (Roy and Cordy 2008) TXL and string comparison (LCS) simian (Harris 2015) line-based string…”
BackgroundMethodInfluentialAn Introduction to Kolmogorov Complexity and Its Applications
Li, M., & Vitányi, P. (2008) · Texts in Computer Science · doi:10.1007/978-0-387-49820-1 · Cited by 1,424
Clone detection using abstract syntax trees
Baxter, I. D., Yahin, A., Moura, L., Sant'Anna, M., & Bier, L. (2002) · Proceedings. International Conference on Software Maintenance (Cat. No. 98CB36272) · doi:10.1109/icsm.1998.738528 · Cited by 1,277 · cited 5×
“A few examples of well-known tree-based tools include CloneDR (Baxter et al. 1998), and Deckard (Jiang et al.”
BackgroundMethodInfluentialSoftware: Practice and Experience
(2006) · Software Practice and Experience · doi:10.1002/(issn)1097-024x · Cited by 1,218
Cilibrasi, R., & Vitanyi, P. M. B. (2005) · IEEE Transactions on Information Theory · doi:10.1109/tit.2005.844059 · Cited by 1,188
Schleimer, S., S, W. D., & Aiken, A. (2003) · Proceedings of the 2003 ACM SIGMOD International Conference on Management of Data · doi:10.1145/872757.872770 · Cited by 1,099
A Taxonomy of Obfuscating Transformations
Collberg, C. S., Thomborson, C. D., & Low, D. (1997) · ResearchSpace (University of Auckland) · Cited by 1,090
DECKARD: Scalable and Accurate Tree-Based Detection of Code Clones
Jiang, L., Misherghi, G., Su, Z., & Glondu, S. (2007) · 29th International Conference on Software Engineering (ICSE'07) · doi:10.1109/icse.2007.30 · Cited by 1,039 · cited 1×
“Examples include finding similar bug fixes (Hartmann et al. 2010), identifying cross-cutting concerns (Bruntink et al. 2005), program comprehension (Maletic and Marcus 2001), code recommendation (Holmes and Murphy 2005), and example extraction (Moreno et al. 2015).”
BackgroundComparison and evaluation of code clone detection techniques and tools: A qualitative approach
Roy, C. K., Cordy, J. R., & Koschke, R. (2009) · Science of Computer Programming · doi:10.1016/j.scico.2009.02.007 · Cited by 1,001 · cited 3×
“Bellon et al. (2007) proposed a framework for comparing and evaluating 6 clone detectors, Roy et al. (2009) evaluated a large set of clone detection tools but only based on results obtained from the tools’ published papers, Hage et al. (2010) compare five plagiarism detectors against 17 code…”
BackgroundMethodCIL: Intermediate Language and Tools for Analysis and Transformation of C Programs
Necula, G. C., McPeak, S., Rahul, S. P., & Weimer, W. (2002) · Lecture Notes in Computer Science · doi:10.1007/3-540-45937-5_16 · Cited by 977 · cited 2×
“…tools are available nowadays both for commercial (e.g. Semantic Designs Inc.’s C obfuscator (Semantic Designs 2016), Stunnix’s obfuscators (Stunnix 2016), Diablo (Maebe and Sutter 2006)) and research purposes (Chow et al. 2001; Schulze and Meyer 2013; Madou et al. 2006; Necula et al. 2002).”
BackgroundMethodA Critique and Improvement of the CL Common Language Effect Size Statistics of McGraw and Wong
Vargha, A., & Delaney, H. D. (2000) · Journal of Educational and Behavioral Statistics · doi:10.3102/10769986025002101 · Cited by 938
Comparison and Evaluation of Clone Detection Tools
Bellon, S., Koschke, R., Antoniol, G., Krinke, J., & Merlo, E. (2007) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2007.70725 · Cited by 782 · cited 2×
“Bellon et al. (2007) proposed a framework for comparing and evaluating 6 clone detectors, Roy et al. (2009) evaluated a large set of clone detection tools but only based on results obtained from the tools’ published papers, Hage et al. (2010) compare five plagiarism detectors against 17 code…”
MethodSelf-citationWatermarking, tamper-proofing, and obfuscation - tools for software protection
Collberg, C. S., & Thomborson, C. (2002) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2002.1027797 · Cited by 735
A comparison of statistical significance tests for information retrieval evaluation
Smucker, M. D., Allan, J., & Carterette, B. (2007) · Proceedings of the sixteenth ACM conference on Conference on information and knowledge management · doi:10.1145/1321440.1321528 · Cited by 718
Identifying similar code with program dependence graphs
Krinke, J. (2002) · Proceedings Eighth Working Conference on Reverse Engineering · doi:10.1109/wcre.2001.957835 · Cited by 616
Self-citationA language independent approach for detecting duplicated code
Ducasse, S., Rieger, M., & Demeyer, S. (1999) · Proceedings IEEE International Conference on Software Maintenance - 1999 (ICSM'99). 'Software Maintenance for Business Change' (Cat. No.99CB36360) · doi:10.1109/icsm.1999.792593 · Cited by 615 · cited 1×
“Other text-based tools with string matching algorithms other than LCS include, but not limited to, Duploc (Ducasse et al. 1999), Simian (Harris 2015), and PMD’s Copy/Paste Detector (CPD) (Dangel A and Pelisse R 2011).”
MethodCP-Miner: finding copy-paste and related bugs in large-scale software code
Li, Z., Lu, S., Myagmar, S., & Zhou, Y. (2006) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2006.28 · Cited by 610 · cited 2×
“Tools that rely on tokens include Sherlock (Joy and Luck 1999), BOSS (Joy et al. 2005), Sim (Gitchell and Tran 1999), YAP3 (Wise 1996), JPlag (Prechelt et al. 2002), CCFinder (Kamiya et al. 2002), CP-Miner (Li et al. 2006), MOSS (Schleimer et al. 2003), Burrows et al. (2007), and the Source Code Similarity Detector Sy…”
BackgroundRoy, C. K., & Cordy, J. R. (2008) · 2008 16th IEEE International Conference on Program Comprehension · doi:10.1109/icpc.2008.41 · Cited by 533 · cited 15×
“Moreover, they used their framework for a comparison limited to three variants of their own clone detector NICAD (Roy and Cordy, 2008).”
MethodInfluentialUsing Slicing to Identify Duplication in Source Code
Komondoor, R., & Horwitz, S. (2001) · Lecture Notes in Computer Science · doi:10.1007/3-540-47764-0_3 · Cited by 527 · cited 2×
“Structural similarity of cloned code can be discovered by using abstract syntax trees as found in CloneDR (Baxter et al, 1998) and Deckard (Jiang et al, 2007a) or by using program dependence graphs (Krinke, 2001; Komondoor and Horwitz, 2001; Liu et al, 2006).”
MethodSourcererCC: scaling code clone detection to big-code
Sajnani, H., Saini, V., Svajlenko, J., Roy, C. K., & Lopes, C. V. (2016) · Proceedings of the 38th International Conference on Software Engineering · doi:10.1145/2884781.2884877 · Cited by 447 · cited 1×
“Anexample is the large-scale token-basedclonedetection tool SourcererCC (Sajnani et al. 2016).”
MethodGPLAG: detection of software plagiarism by program dependence graph analysis
Liu, C., Chen, C., Han, J., & Yu, P. S. (2006) · Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1150402.1150522 · Cited by 422 · cited 1×
“Algorithms for graph comparison are mostly NP-complete (Liu et al. 2006; Crussell et al. 2012; Krinke 2001; Chae et al. 2013).”
MethodUsing structural context to recommend source code examples
Holmes, R., & Murphy, G. C. (2005) · Proceedings of the 27th international conference on Software engineering - ICSE '05 · doi:10.1145/1062455.1062491 · Cited by 379
“Cloning considered harmful” considered harmful: patterns of cloning in software
Kapser, C. J., & Godfrey, M. W. (2008) · Empirical Software Engineering · doi:10.1007/s10664-008-9076-6 · Cited by 365
Plagiarism in programming assignments
Joy, M., & Luck, M. (1999) · IEEE Transactions on Education · doi:10.1109/13.762946 · Cited by 299
YAP3: improved detection of similarities in computer program and other texts
Wise, M. J. (1996) · Proceedings of the twenty-seventh SIGCSE technical symposium on Computer science education · doi:10.1145/236452.236525 · Cited by 294 · cited 2×
“Tools that rely on tokens include Sherlock (Joy and Luck 1999), BOSS (Joy et al. 2005), Sim (Gitchell and Tran 1999), YAP3 (Wise 1996), JPlag (Prechelt et al. 2002), CCFinder (Kamiya et al. 2002), CP-Miner (Li et al. 2006), MOSS (Schleimer et al. 2003), Burrows et al. (2007), and the Source Code Similarity Detector Sy…”
MethodAchieving accuracy and scalability simultaneously in detecting application clones on Android markets
Chen, K., Liu, P., & Zhang, Y. (2014) · Proceedings of the 36th International Conference on Software Engineering · doi:10.1145/2568225.2568286 · Cited by 287 · cited 4×
“It has several benefits including recovery of lost source code from compiled artefacts such as binary or bytecode, reverse engineering, finding similar applications (Chen et al. 2014).”
BackgroundMethodInfluentialThe boss online submission and assessment system
Joy, M., Griffiths, N., & Boyatt, R. (2005) · Journal on Educational Resources in Computing · doi:10.1145/1163405.1163407 · Cited by 282 · cited 2×
“Tools that rely on tokens include Sherlock (Joy and Luck 1999), BOSS (Joy et al. 2005), Sim (Gitchell and Tran 1999), YAP3 (Wise 1996), JPlag (Prechelt et al. 2002), CCFinder (Kamiya et al. 2002), CP-Miner (Li et al. 2006), MOSS (Schleimer et al. 2003), Burrows et al. (2007), and the Source Code Similarity Detector Sy…”
MethodWhat would other programmers do: suggesting solutions to error messages
Hartmann, B., MacDougall, D., Brandt, J., & Klemmer, S. R. (2010) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1753326.1753478 · Cited by 280
SourcererCC: Scaling Code Clone Detection to Big Code
Sajnani, H., Saini, V., Svajlenko, J., Roy, C. K., & Lopes, C. V. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1512.06448 · Cited by 280 · cited 1×
“Anexample is the large-scale token-basedclonedetection tool SourcererCC (Sajnani et al. 2016).”
MethodAttack of the Clones: Detecting Cloned Applications on Android Markets
Crussell, J., Gibler, C., & Chen, H. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-33167-1_3 · Cited by 270 · cited 2×
“Algorithms for graph comparison are mostly NP-complete (Liu et al. 2006; Crussell et al. 2012; Krinke 2001; Chae et al. 2013).”
BackgroundMethod"Cloning Considered Harmful" Considered Harmful
Kapser, C., & Godfrey, M. (2006) · 2006 13th Working Conference on Reverse Engineering · doi:10.1109/wcre.2006.1 · Cited by 251 · cited 1×
“Since boiler-plate code is inherently different from pervasively modified code and normally found in software development (Kapser and Godfrey, 2006), the findings give a guideline to choosing the right tools/techniques when measuring code similarity in the presence of boiler-plate code.”
MethodGöde, N., & Koschke, R. (2009) · 2009 13th European Conference on Software Maintenance and Reengineering · doi:10.1109/csmr.2009.20 · Cited by 222 · cited 2×
“Simian (Harris 2015) is a pure, text-based, clone detection tool relying on text line comparison with a capability for checking basic code modifications, e.g. identifier renaming. iClones (Göde and Koschke 2009) performs token-based incremental clone detection over several revisions of a program.”
MethodLuo, L., Ming, J., Wu, D., Liu, P., & Zhu, S. (2014) · Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering · doi:10.1145/2635868.2635900 · Cited by 220 · cited 3×
“Luo et al. (2014) use GCC/G++ with different optimisation options to generate 10 different binary versions of the same program.”
BackgroundMethodSupporting Program Comprehension Using Semantic and Structural Information
Maletic, J. I., & Marcus, A. (2001) · Cited by 215
An algorithmic approach to the detection and prevention of plagiarism
Ottenstein, K. J. (1976) · ACM SIGCSE Bulletin · doi:10.1145/382222.382462 · Cited by 208 · cited 3×
“Early approaches to detect software similarity (Ottenstein, 1976; Donaldson et al, 1981; Grier, 1981; Berghel and Sallach, 1984; Faidhi and Robinson, 1987) are based on metrics or software measures.”
MethodDeobfuscation: Reverse Engineering Obfuscated Code
Udupa, S. K., Debray, S. K., & Madou, M. (2006) · 12th Working Conference on Reverse Engineering (WCRE'05) · doi:10.1109/wcre.2005.13 · Cited by 204
Context-based detection of clone-related bugs
Jiang, L., Su, Z., & Chiu, E. (2007) · Proceedings of the the 6th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on The foundations of software engineering · doi:10.1145/1287624.1287634 · Cited by 202 · cited 9×
“To make sure that we get the most accurate similarity calculation for Deckard and other clone detectors, we also cover the most fine-grained level of source code: characters.”
BackgroundMethodInfluentialFaidhi, J. A. W., & Robinson, S. K. (1987) · Computers & Education · doi:10.1016/0360-1315(87)90042-x · Cited by 198 · cited 1×
“Early approaches to detect software similarity (Ottenstein 1976; Donaldson et al. 1981; Grier 1981; Berghel and Sallach 1984; Faidhi and Robinson 1987) are based on metrics or software measures.”
Decompilation of binary programs
Cifuentes, C., & Gough, K. J. (1995) · Software: Practice and Experience · doi:10.1002/spe.4380250706 · Cited by 194 · cited 2×
“…several applications including recovery of lost source code, migrating a system to another platform, upgrading an old program into a newer programming language, restructuring poorly-written code, finding bugs or malicious code in binary programs, and program validation (Cifuentes and Gough 1995).”
BackgroundA Mutation/Injection-Based Automatic Framework for Evaluating Code Clone Detection Tools
Roy, C. K., & Cordy, J. R. (2009) · 2009 International Conference on Software Testing, Verification, and Validation Workshops · doi:10.1109/icstw.2009.18 · Cited by 193 · cited 4×
“However, whilst they usually have no problem in the presence of local or confined modifications, pervasive modifications that transform whole files remain a challenge (Roy and Cordy 2009).”
BackgroundMethodInfluentialRepairing Programs with Semantic Code Search (T)
Ke, Y., Stolee, K. T., Goues, C. L., & Brun, Y. (2015) · 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2015.60 · Cited by 192 · cited 1×
“This situation occurs when performing code search for automated software repair (Ke et al. 2015).”
An Approach to Source-Code Plagiarism Detection and Investigation Using Latent Semantic Analysis
Cosma, G., & Joy, M. (2012) · IEEE Transactions on Computers · doi:10.1109/tc.2011.223 · Cited by 176
Protection of software-based survivability mechanisms
Wang, C., Davidson, J., Hill, J., & Knight, J. (2002) · Proceedings International Conference on Dependable Systems and Networks · doi:10.1109/dsn.2001.941405 · Cited by 173
Static test case prioritization using topic models
Thomas, S. W., Hemmati, H., Hassan, A. E., & Blostein, D. (2012) · Empirical Software Engineering · doi:10.1007/s10664-012-9219-7 · Cited by 171 · cited 1×
“We choose Vargha and Delaney’s A12 measure because it is robust with respect to the shape of the distributions being compared (Thomas et al. 2014).”
MethodAnDarwin: Scalable Detection of Semantically Similar Android Applications
Crussell, J., Gibler, C., & Chen, H. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-40203-6_11 · Cited by 169 · cited 2×
“…cloned and plagiarised programs (especially mobile applications) based on compiled code (Chae et al. 2013; Chen et al. 2014; Gibler et al. 2013; Crussell et al. 2012, 2013; Tian et al. 2014; Tamada et al. 2004; Myles and Collberg 2004; Hi et al. 2009; Zhang et al. 2012, 2014; McMillan et al.…”
BackgroundViewDroid: towards obfuscation-resilient mobile application repackaging detection
Zhang, F., Huang, H., Zhu, S., Wu, D., & Liu, P. (2014) · Proceedings of the 2014 ACM conference on Security and privacy in wireless & mobile networks · doi:10.1145/2627393.2627395 · Cited by 167
Evaluating clone detection tools for use during preventative maintenance
Burd, E., & Bailey, J. (2003) · IEEE International Working Conference on Source Code Analysis and Manipulation (SCAM) · doi:10.1109/scam.2002.1134103 · Cited by 164
Sim: a utility for detecting similarity in computer programs
Gitchell, D., & Tran, N. (1999) · The proceedings of the thirtieth SIGCSE technical symposium on Computer science education · doi:10.1145/299649.299783 · Cited by 162 · cited 4×
“…(Prechelt et al. 2002) tokens, Karp Rabin matching, Greedy String Tiling plaggie (Ahtiainen et al. 2006) N/A (not disclosed) sherlock (Pike R and Loki 2002) digital signatures simjava (Gitchell and Tran 1999) tokens and string alignment simtext (Gitchell and Tran 1999) tokens and string alignment”
BackgroundMethodInfluentialOn the use of clone detection for identifying crosscutting concern code
Bruntink, M., van Deursen, A., van Engelen, R., & Tourwe, T. (2005) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2005.114 · Cited by 158 · cited 1×
“Examples include finding similar bug fixes (Hartmann et al. 2010), identifying cross-cutting concerns (Bruntink et al. 2005), program comprehension (Maletic and Marcus 2001), code recommendation (Holmes and Murphy 2005), and example extraction (Moreno et al. 2015).”
BackgroundSupporting program comprehension using semantic and structural information
Maletic, J. I., & Marcus, A. (2005) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2001.919085 · Cited by 150 · cited 3×
“’s C obfuscator (Semantic Designs 2016), Stunnix’s obfuscators (Stunnix 2016), Diablo (Maebe and Sutter 2006)) and research purposes (Chow et al.”
BackgroundMethodSearching for better configurations: a rigorous approach to clone evaluation
Wang, T., Harman, M., Jia, Y., & Krinke, J. (2013) · Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering · doi:10.1145/2491411.2491420 · Cited by 147 · cited 3×
“We modified and integrated the GCF Converter (Wang et al. 2013) to convert clone reports generated by unsupported clone detectors into GCF format.”
MethodResultInfluentialSelf-citationTowards a Definition of Source-Code Plagiarism
Cosma, G., & Joy, M. (2008) · IEEE Transactions on Education · doi:10.1109/te.2007.906776 · Cited by 146
Detecting Software Theft via Whole Program Path Birthmarks
Myles, G., & Collberg, C. S. (2004) · Lecture notes in computer science · doi:10.1007/978-3-540-30144-8_34 · Cited by 145 · cited 2×
“In the last few years, there have been several studies to discover cloned and plagiarised programs (especially mobile applications) based on compiled code (Chae et al, 2013; Chen et al, 2014; Gibler et al, 2013; Crussell et al, 2013; Tian et al, 2014; Tamada et al, 2004; Myles and Collberg, 2004; Lim et al, 2009; Zhan…”
BackgroundSpotting working code examples
Keivanloo, I., Rilling, J., & Zou, Y. (2014) · Proceedings of the 36th International Conference on Software Engineering · doi:10.1145/2568225.2568292 · Cited by 145 · cited 1×
“Another example is searching for working code examples (Keivanloo et al. 2014) when one wants to pick only the top ranked solution.”
BackgroundSandmark-A tool for software protection research
Collberg, C., Myles, G. R., & Huntwork, A. (2003) · IEEE Security & Privacy · doi:10.1109/msecp.2003.1219058 · Cited by 126 · cited 5×
“Keywords Empirical study · Code similarity measurement · Clone detection · Plagiarism detection · Parameter optimisation”
BackgroundMethodInfluentialMoreno, L., Bavota, G., Di Penta, M., Oliveto, R., & Marcus, A. (2015) · 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering · doi:10.1109/icse.2015.98 · Cited by 125 · cited 1×
“Examples include finding similar bug fixes (Hartmann et al. 2010), identifying cross-cutting concerns (Bruntink et al. 2005), program comprehension (Maletic and Marcus 2001), code recommendation (Holmes and Murphy 2005), and example extraction (Moreno et al. 2015).”
BackgroundAn Approach to the Obfuscation of Control-Flow of Sequential Computer Programs
Chow, S., Gu, Y. X., Johnson, H., & Zakharov, V. A. (2001) · Lecture notes in computer science · doi:10.1007/3-540-45439-x_10 · Cited by 124 · cited 2×
“…tools are available nowadays both for commercial (e.g. Semantic Designs Inc.’s C obfuscator (Semantic Designs 2016), Stunnix’s obfuscators (Stunnix 2016), Diablo (Maebe and Sutter 2006)) and research purposes (Chow et al. 2001; Schulze and Meyer 2013; Madou et al. 2006; Necula et al. 2002).”
BackgroundMethodEvaluating Modern Clone Detection Tools
Svajlenko, J., & Roy, C. K. (2014) · 2014 IEEE International Conference on Software Maintenance and Evolution · doi:10.1109/icsme.2014.54 · Cited by 120 · cited 2×
“on the accuracy of clone detection tools (Bellon et al, 2007; Roy et al, 2009; Svajlenko and Roy, 2014) and of plagiarism detection tools (Hage et al, 2010), were mainly focused on a single technique or tool, or on a single domain.”
BackgroundMethodA Source Code Similarity System for Plagiarism Detection
Duric, Z., & Gašević, D. (2012) · The Computer Journal · doi:10.1093/comjnl/bxs018 · Cited by 118 · cited 3×
“Many tools transform source code into an intermediate representation such as tokens and apply similarity measurement on them (Plague (Whale, 1990), Sherlock (Joy et al, 2005), Sim (Gitchell andTran, 1999), YAP3 (Wise, 1996), JPlag (Prechelt et al, 2002), CCFinder (Kamiya et al, 2002), CP-Miner (Li et al, 2006), iClone…”
BackgroundMethodDetecting similar software applications
McMillan, C., Grechanik, M., & Poshyvanyk, D. (2012) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2012.6227178 · Cited by 116
BigCloneEval: A Clone Detection Tool Evaluation Framework with BigCloneBench
Svajlenko, J., & Roy, C. K. (2016) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme.2016.62 · Cited by 113 · cited 1×
“…five clone detectors for preventive maintenance tasks, Biegel et al. (2011) compare three code similarity measures to identify code that needs refactoring, Svajlenko and Roy (2016) developed and used a clone evaluation framework called BigCloneEval to evaluate 10 state-of-the-art clone detectors.”
MethodA tool that detects plagiarism in Pascal programs
Grier, S. (1981) · Proceedings of the twelfth SIGCSE technical symposium on Computer science education - SIGCSE '81 · doi:10.1145/800037.800954 · Cited by 112 · cited 1×
“Early approaches to detect software similarity (Ottenstein 1976; Donaldson et al. 1981; Grier 1981; Berghel and Sallach 1984; Faidhi and Robinson 1987) are based on metrics or software measures.”
Detecting similar software applications
McMillan, C., Grechanik, M., & Poshyvanyk, D. (2012) · Cited by 112
Efficient plagiarism detection for large code repositories
Burrows, S., Tahaghoghi, S. M. M., & Zobel, J. (2006) · Software Practice and Experience · doi:10.1002/spe.750 · Cited by 109 · cited 3×
“2013), and giving programming assignments (Burrows et al. 2007; Schleimer et al. 2003).”
BackgroundDynamic Software Birthmarks to Detect the Theft of Windows Applications
Tamada, H., Okamoto, K., Nakamura, M., Monden, A., & Matsumoto, K. (2004) · Cited by 107
Identification of Program Similarity in Large Populations
Whale, G. (1990) · The Computer Journal · doi:10.1093/comjnl/33.2.140 · Cited by 101 · cited 3×
“One of the widely-used string similarity methods is to find a longest common subsequence (LCS) which is adopted by the NiCad clone detector (Roy and Cordy 2008), Plague (Whale 1990), the first version of YAP (Wise 1992), and CoP (Luo et al. 2014).”
BackgroundMethodStack Overflow in Github: Any Snippets There?
Yang, D., Martins, P., Saini, V., & Lopes, C. V. (2017) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr.2017.13 · Cited by 101
Moreno, L., Bavota, G., Penta, M. D., et al. (2015) · Open Research (University of Surrey) · doi:10.5555/2818754.2818860 · Cited by 93
Ceccato, M., Penta, M. D., Falcarin, P., Ricca, F., Torchiano, M., & Tonella, P. (2013) · Empirical Software Engineering · doi:10.1007/s10664-013-9248-x · Cited by 89 · cited 3×
“Algorithms for graph comparison are mostly NP-complete (Liu et al. 2006; Crussell et al. 2012; Krinke 2001; Chae et al. 2013).”
BackgroundMethodDonaldson, J. L., Lancaster, A.-M., & Sposato, P. H. (1981) · Proceedings of the twelfth SIGCSE technical symposium on Computer science education - SIGCSE '81 · doi:10.1145/800037.800955 · Cited by 88 · cited 1×
“Early approaches to detect software similarity (Ottenstein 1976; Donaldson et al. 1981; Grier 1981; Berghel and Sallach 1984; Faidhi and Robinson 1987) are based on metrics or software measures.”
Detection of similarities in student programs: YAP'ing may be preferable to plague'ing
Wise, M. J. (1992) · Proceedings of the twenty-third SIGCSE technical symposium on Computer science education · doi:10.1145/134510.134564 · Cited by 87 · cited 2×
“One of the widely-used string similarity methods is to find a longest common subsequence (LCS) which is adopted by the NiCad clone detector (Roy and Cordy 2008), Plague (Whale 1990), the first version of YAP (Wise 1992), and CoP (Luo et al. 2014).”
MethodThe Issue of (Software) Plagiarism: A Student View
Chuda, D., Navrat, P., Kovacova, B., & Humay, P. (2012) · IEEE Transactions on Education · doi:10.1109/te.2011.2112768 · Cited by 86
Plaggie: GNU-licensed source code plagiarism detection engine for Java exercises
Ahtiainen, A., Surakka, S., & Rahikainen, M. (2006) · Proceedings of the 6th Baltic Sea conference on Computing education research: Koli Calling 2006 · doi:10.1145/1315803.1315831 · Cited by 84 · cited 2×
“Plaggie’s detection (Ahtiainen et al. 2006) method is not public but claims to have the same functionalities as JPlag.”
BackgroundMethodClone evolution: a systematic review
Pate, J. R., Tairas, R., & Kraft, N. A. (2011) · Journal of Software Evolution and Process · doi:10.1002/smr.579 · Cited by 83 · cited 1×
“Nevertheless, they also represent code clones that are repeatedly modified over time during software evolution (Pate et al. 2013), and source code before and after refactoring activities (Fowler 2013).”
BackgroundGibler, C., Stevens, R., Crussell, J., Chen, H., Zang, H., & Choi, H. (2013) · International Conference on Mobile Systems, Applications, and Services · doi:10.1145/2462456.2464461 · Cited by 81 · cited 2×
“…studies to discover cloned and plagiarised programs (especially mobile applications) based on compiled code (Chae et al. 2013; Chen et al. 2014; Gibler et al. 2013; Crussell et al. 2012, 2013; Tian et al. 2014; Tamada et al. 2004; Myles and Collberg 2004; Hi et al. 2009; Zhang et al. 2012,…”
BackgroundThe effectiveness of source code obfuscation: An experimental assessment
Ceccato, M., Penta, M. D., Nagra, J., Falcarin, P., Ricca, F., Torchiano, M., & Tonella, P. (2009) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc.2009.5090041 · Cited by 79
Toward a Taxonomy of Clones in Source Code: A Case Study
Kapser, C. J., & Godfrey, M. W. (2003) · Cited by 75
Software plagiarism detection: a graph-based approach
Chae, D.-K., Ha, J., Kim, S.-W., Kang, B., & Im, E. G. (2013) · Proceedings of the 22nd ACM international conference on Information & Knowledge Management · doi:10.1145/2505515.2507848 · Cited by 66 · cited 3×
“Algorithms for graph comparison are mostly NP-complete (Liu et al. 2006; Crussell et al. 2012; Krinke 2001; Chae et al. 2013).”
BackgroundMethodA first step towards algorithm plagiarism detection
Zhang, F., Jhi, Y.-C., Wu, D., Liu, P., & Zhu, S. (2012) · Proceedings of the 2012 International Symposium on Software Testing and Analysis · doi:10.1145/2338965.2336767 · Cited by 63
A comparison of plagiarism detection tools
Hage, J., & Rademaker, P. (2010) · Cited by 61
Type-Based Decompilation (or Program Reconstruction via Type Reconstruction)
Mycroft, A. (1999) · Lecture notes in computer science · doi:10.1007/3-540-49099-x_14 · Cited by 58 · cited 1×
“Many decompilers have been invented in the literature for various programming languages (Cifuentes and Gough 1995; Proebsting and Watterson 1997; Desnos and Gueguen 2011; Mycroft 1999; Breuer and Bowen 1994).”
BackgroundLanguage-Independent Clone Detection Applied to Plagiarism Detection
Brixtel, R., Fontaine, M., Lesner, B., Bazin, C., & Robbes, R. (2010) · 2010 10th IEEE Working Conference on Source Code Analysis and Manipulation · doi:10.1109/scam.2010.19 · Cited by 56 · cited 1×
“…investigated the code transformations offered by Artifice and ProGuard and found that they cover changes commonly found in both code cloning and code plagiarism as reported by Roy and Cordy (2009), Schulze and Meyer (2013), Duric and Gasevic (2013), Joy and Luck (1999), and Brixtel et al. (2010).”
MethodFinding Plagiarisms among a Set of Programs with JPlag
Malpohl, G., Prechelt, L., & Philippsen, M. (2002) · Verlag der Technischen Universität Graz · doi:10.3217/jucs-008-11-1016 · Cited by 56
Using compilation/decompilation to enhance clone detection
Ragkhitwetsagul, C., & Krinke, J. (2017) · IEEE International Workshop on Software Clones (IWSC) · doi:10.1109/iwsc.2017.7880502 · Cited by 55 · cited 3×
“An empirical study (Ragkhitwetsagul and Krinke, 2017) shows that, compared to clones in the original versions, additional clones were found after compilation/decompilation in three real-world software projects.”
BackgroundResultSelf-citationMeasurements of program similarity in identical task environments
Berghel, H. L., & Sallach, D. L. (1984) · ACM SIGPLAN Notices · doi:10.1145/988241.988245 · Cited by 54 · cited 1×
“Early approaches to detect software similarity (Ottenstein 1976; Donaldson et al. 1981; Grier 1981; Berghel and Sallach 1984; Faidhi and Robinson 1987) are based on metrics or software measures.”
Enhancing Source-Based Clone Detection Using Intermediate Representation
Selim, G. M. K., Foo, K. C., & Zou, Y. (2010) · Working Conference on Reverse Engineering (WCRE) · doi:10.1109/wcre.2010.33 · Cited by 54
Protection of Software-based Survivability Mechanisms
Wang, C., Davidson, J. W., Hill, J., & KNIGHT, J. C. (2001) · Defense Technical Information Center (DTIC) · doi:10.21236/ada466288 · Cited by 49
Software Bertillonage: Determining the provenance of software development artifacts
Davies, J., Germán, D. M., Godfrey, M. W., & Hindle, A. (2012) · Empirical Software Engineering · doi:10.1007/s10664-012-9199-7 · Cited by 47
A method for detecting the theft of Java programs through analysis of the control flow information
Lim, H.-I., Park, H., Choi, S., & Han, T. (2009) · Information and Software Technology · doi:10.1016/j.infsof.2009.04.011 · Cited by 46
Threshold-free code clone detection for a large-scale heterogeneous Java repository
Keivanloo, I., Zhang, F., & Zou, Y. (2015) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner.2015.7081830 · Cited by 45 · cited 1×
“Unfortunately, the metrics-based approaches have been found empirically to be less effective in comparison with other, newer approaches (Kapser and Godfrey 2003).”
ResultFrom Whence It Came: Detecting Source Code Clones by Analyzing Assembler
Davis, I. J., & Godfrey, M. W. (2010) · Working Conference on Reverse Engineering (WCRE) · doi:10.1109/wcre.2010.35 · Cited by 44 · cited 1×
“There are studies that try to enhance performance of clone detectors by looking for more clones from the code’s intermediate representation such as Jimple code (Selim et al, 2010), bytecode (Chen et al, 2014;Kononenko et al, 2014), or assembler code (Davis and Godfrey, 2010).”
BackgroundLOCO: an interactive code (De)obfuscation tool
Madou, M., Van Put, L., & De Bosschere, K. (2006) · Proceedings of the 2006 ACM SIGPLAN symposium on Partial evaluation and semantics-based program manipulation · doi:10.1145/1111542.1111566 · Cited by 43 · cited 3×
“Loco (Madou et al. 2006) is a binary obfuscator capable of performing obfuscation using control flow flattening and opaque predicates on selected fragments of code.”
BackgroundMethodA tool that detects plagiarism in Pascal programs
Grier, S. (1981) · ACM SIGCSE Bulletin · doi:10.1145/953049.800954 · Cited by 41
Obfuscating Java: The Most Pain for the Least Gain
Batchelder, M., & Hendren, L. J. (2007) · Springer eBooks · doi:10.1007/978-3-540-71229-9_7 · Cited by 39 · cited 1×
“Because Java byte code is comparatively high-level and easy to decompile, obfuscation of Java bytecode has focused on preventing decompilation (Batchelder and Hendren 2007) whilst decompilers like Krakatoa (Proebsting and Watterson Empir Software Eng (2018) 23:2 –2 464 519 2511”
BackgroundSimilarity of Source Code in the Presence of Pervasive Modifications
Ragkhitwetsagul, C., Krinke, J., & Clark, D. (2016) · IEEE International Working Conference on Source Code Analysis and Manipulation (SCAM) · doi:10.1109/scam.2016.13 · Cited by 39
Self-citationKrakatoa: decompilation in java (dose bytecode reveal source?)
Proebsting, T. A., & Watterson, S. A. (1997) · USENIX Conference on Object-Oriented Technologies and Systems · Cited by 36
Comparison of similarity metrics for refactoring detection
Biegel, B., Soetens, Q. D., Hornig, W., Diehl, S., & Demeyer, S. (2011) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/1985441.1985452 · Cited by 36 · cited 1×
“…five plagiarism detectors against 17 code modifications, Burd and Bailey (2002) compare five clone detectors for preventive maintenance tasks, Biegel et al. (2011) compare three code similarity measures to identify code that needs refactoring, Svajlenko and Roy (2016) developed and used a…”
MethodDecompilation: the enumeration of types and grammars
Breuer, P., & Bowen, J. P. (1994) · ACM Transactions on Programming Languages and Systems · doi:10.1145/186025.186093 · Cited by 31 · cited 2×
“Many decompilers have been invented in the literature for various programming languages (Cifuentes and Gough, 1995; Proebsting and Watterson, 1997; Desnos and Gueguen, 2011; Mycroft, 1999; Breuer and Bowen, 1994).”
BackgroundDetection of similarities in student programs: YAP'ing may be preferable to plague'ing
Wise, M. J. (1992) · ACM SIGCSE Bulletin · doi:10.1145/135250.134564 · Cited by 28
On the robustness of clone detection to code obfuscation
Schulze, S., & Meyer, D. (2013) · IEEE International Workshop on Software Clones (IWSC) · doi:10.1109/iwsc.2013.6613045 · Cited by 24 · cited 5×
“The work that is closest to ours is the empirical study of the efficiency of current detection tools against code obfuscation (Schulze and Meyer 2013).”
BackgroundMethodInfluentialCompiling Clones: What Happens?
Kononenko, O., Zhang, C., & Godfrey, M. W. (2014) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme.2014.78 · Cited by 19
Plagiarism detection for multithreaded software based on thread-aware software birthmarks
Tian, Z., Zheng, Q., Liu, T., Fan, M. Y., Zhang, X., & Yang, Z. (2014) · International Conference on Program Comprehension · doi:10.1145/2597008.2597143 · Cited by 18 · cited 1×
“…programs (especially mobile applications) based on compiled code (Chae et al. 2013; Chen et al. 2014; Gibler et al. 2013; Crussell et al. 2012, 2013; Tian et al. 2014; Tamada et al. 2004; Myles and Collberg 2004; Hi et al. 2009; Zhang et al. 2012, 2014; McMillan et al. 2012; Luo et al. 2014).”
BackgroundOn the robustness of clone detection to code obfuscation
Schulze, S., & Meyer, D. (2013) · International Workshop on Software Clones · doi:10.5555/2662708.2662722 · Cited by 18
Beitzel, S. M., Jensen, E. C., & Frieder, O. (2016) · Springer eBooks · doi:10.1007/978-1-4899-7993-3_491-2 · Cited by 1
Research works printed but not linked: 2
- 73. Manning CD, Raghavan P, Schutze H (2009) An introduction toinformation retrieval, vol 21. Cambridge University Press, Cambridge
- 78. Nachenberg C (1996) Understanding and managing polymorphic viruses. The Symantec Enterprise Papers 30:16
Cited by
A Systematic Review on Code Clone Detection
Ain, Q. U., Butt, W. H., Anwar, M. W., Azam, F., & Maqbool, B. (2019) · IEEE Access · doi:10.1109/access.2019.2918202 · Cited by 165
“[76] compare the code similarity analyzers.”
BackgroundMethodInfluentialToxic Code Snippets on Stack Overflow
Ragkhitwetsagul, C., Krinke, J., Paixão, M. H. E., Bianco, G., & Oliveto, R. (2019) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2900307 · Cited by 136
“We are aware of effects of configurations to clone detection results and the importance of searching for optimised configurations in empirical clone studies [40], [41], [51], [52], [53].”
BackgroundSelf-citationComplement or Substitute? How AI Increases the Demand for Human SkillsPreprint
Mäkelä, E., & Stephany, F. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5153230 · Cited by 115
Nasrabadi, M. Z., Parsa, S. A., Ramezani, M., Roy, C. K., & Ekhtiarzadeh, M. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111796 · Cited by 129
Detecting LLM-Generated Text in Computing Education: Comparative Study for ChatGPT Cases
Orenstrakh, M. S., Karnalim, O., Suárez, C. A., & Liut, M. (2024) · 2024 IEEE 48th Annual Computers, Software, and Applications Conference (COMPSAC) · doi:10.1109/compsac61105.2024.00027 · Cited by 79
“In the context of plagiarism and collusion, automated detectors nullify common alterations that can be done without understanding the content [24, 43] and remove contents that are not evident for raising suspicion [48].”
Background
Metadata
- Authors
- Ragkhitwetsagul, C., Krinke, J., & Clark, D.
- Affiliations
- University College London (GB)
- Corresponding author
- Ragkhitwetsagul, Chaiyong
- Krinke, Jens
- Clark, David
- Venue
- Empirical Software Engineering
- Published
- 2017-10-25
- Volume
- 23
- Issue
- 4
- Pages
- 2464-2519
- ISSN
- 1382-3256 (ISSN-L)
- 1573-7616
- OpenAlex
- W2766078762
- Semantic Scholar
- ed1b5e97277bdb779ee015420f9e188a105f3df0
- Cited by
- 122
- References
- 110
- Citations by year
- 2026: 3 · 2025: 11 · 2024: 17 · 2023: 14 · 2022: 22 · 2021: 17 · 2020: 20 · 2019: 14 · 2018: 4
- FWCI
- 19.35
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Empirical Software Engineering (publishedVersion, cc-by)
- UCL Discovery (University College London) (submittedVersion)
- Funders
- Mahidol University
- Faculty of Information and Communication Technology, Mahidol University
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Social Sciences › Education › Academic integrity and plagiarism
- Physical Sciences › Computer Science › Hardware and Architecture › Physical Unclonable Functions (PUFs) and Hardware Security
- Keywords
- code clone detection, software plagiarism detection, code modification, obfuscation, boilerplate code, Java source code, false positive reduction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.