Towards a Big Data Curated Benchmark of Inter-project Code Clones
Abstract
Recently, new applications of code clone detection and search have emerged that rely upon clones detected across thousands of software systems. Big data clone detection and search algorithms have been proposed as an embedded part of these new applications. However, there exists no previous benchmark data for evaluating the recall and precision of these emerging techniques. In this paper, we present a Big Data clone detection benchmark that consists of known true and false positive clones in a Big Data inter-project Java repository. The benchmark was built by mining and then manually checking clones of ten common functionalities. The benchmark contains six million true positive clones of different clone types: Type-1, Type-2, Type-3 and Type-4, including various strengths of Type-3 similarity (strong, moderate, weak). These clones were found by three judges over 216 hours of manual validation efforts. We show how the benchmark can be used to measure the recall and precision of clone detection techniques.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
AgenticFlict is the first large-scale dataset of textual merge conflicts in Agentic PRs.
“To the best of our knowledge, AgenticFlict is the first largescale dataset of textual merge conflicts in Agentic PRs.”
From AgenticFlict: A Large-Scale Dataset of Merge Conflicts in AI Coding Agent Pull Requests on GitHub (2026) · cites this paper as [49]
ConflictBench is a dedicated benchmark specifically designed to evaluate merge tools.
“ConflictBench [49] was introduced as a dedicated benchmark specifically designed to evaluate merge tools.”
From AgenticFlict: A Large-Scale Dataset of Merge Conflicts in AI Coding Agent Pull Requests on GitHub (2026) · cites this paper as [49]
Cited by
Sorted by citations
Large Language Models for Software Engineering: A Systematic Literature Review
Hou, X., Zhao, Y., Liu, Y., Yang, Z., Wang, K., Li, L., Luo, X., Lo, D., Grundy, J., & Wang, H. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3695988 · Cited by 1,029 · see citing works
UniXcoder: Unified Cross-Modal Pre-training for Code Representation
Guo, D., Lu, S., Duan, N., Wang, Y., Zhou, M., & Yin, J. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.499 · Cited by 673 · see citing works
Program Synthesis with Large Language Models
Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07732 · Cited by 388 · see citing works
“…have been proposed for software engineering tasks including code completion [Raychev et al., 2016, Allamanis and Sutton, 2013b], clone detection [Svajlenko et al., 2014], code search [Husain et al., 2019], predicting readable names to describe functions [Allamanis et al., 2016], and generating…”
BackgroundMethodEvaluating clone detection tools with BigCloneBench
Svajlenko, J., & Roy, C. K. (2015) · 2015 IEEE International Conference on Software Maintenance and Evolution (ICSME) · doi:10.1109/icsm.2015.7332459 · Cited by 172 · see citing works
Self-citationAn extensive study on pre-trained models for program understanding and generation
Zeng, Z., Tan, H., Zhang, H., Li, J., Zhang, Y., & Zhang, L. (2022) · Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3533767.3534390 · Cited by 153 · see citing works
A Systematic Literature Review on the Use of Deep Learning in Software Engineering Research
Watson, C., Cooper, N., Palacio, D. N., Moran, K., & Poshyvanyk, D. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3485275 · Cited by 137 · see citing works
Zhou, X., Weyssow, M., Widyasari, R., Zhang, T., He, J., Lyu, Y., Chang, J., Zhang, B., Huang, D., & Lo, D. (2025) · arXiv · doi:10.48550/arxiv.2502.06215 · Cited by 3 · see citing works
A Blockchain-Based Copyright Management Scheme for Open-Source Software and Hardware Designs
Jin, Y., & Ling, L. (2025) · Tsinghua Science and Technology · doi:10.26599/tst.2023.9010112 · Cited by 1 · see citing works
AgenticFlict: A Large-Scale Dataset of Merge Conflicts in AI Coding Agent Pull Requests on GitHub
Ogenrwot, D., & Businge, J. (2026) · Proceedings of the 3rd ACM International Conference on AI-Powered Software · doi:10.1145/3805760.3814923
“This approach follows established practices in large-scale mining of GitHub data [17, 26, 48] and aligns with GitHub’s Terms of Service governing code redistribution [21].”
Demonstration Attack against In-Context Learning for Code Intelligence
Ge, Y., Sun, W., Lou, Y., Fang, C., Zhang, Y., Li, Y., Zhang, X., Liu, Y., Zhao, Z., & Chen, Z. (2024) · arXiv · doi:10.48550/arxiv.2410.02841
“For the clone detection task, we use the BigCloneBench dataset introduced by Roy et al. [40].”
Method
- Authors
- Svajlenko, J., Islam, J. F., Keivanloo, I., Roy, C. K., & Mia, M. M.
- Affiliations
- University of Saskatchewan (CA)
- Queen's University (CA)
- Venue
- 2014 IEEE International Conference on Software Maintenance and Evolution
- Publisher
- IEEE
- Published
- 2014-09
- Pages
- 476-480
- ISSN
- 1063-6773 (ISSN-L)
- OpenAlex
- W2065053490
- Cited by
- 339
- References
- 0
- Citations by year
- 2026: 26 · 2025: 45 · 2024: 42 · 2023: 58 · 2022: 49 · 2021: 27 · 2020: 23 · 2019: 27 · 2018: 11 · 2017: 12 · 2016: 13 · 2015: 6
- FWCI
- 8.82
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE International Conference on Software Maintenance (ICSM) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Social Sciences › Decision Sciences › Management Science and Operations Research › Data Quality and Management
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Keywords
- code clone detection, benchmark, big data, manual validation, recall and precision evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
