CoChat
Sign Up
Cited by 158Software Engineering ResearchSoftware Testing and Debugging TechniquesSoftware Engineering Techniques and Practices

Deep learning similarities from different representations of source code

Open access
Tufano, Michele, Watson, Cody, Bavota, Gabriele, Di Penta, Massimiliano, White, Martin, Poshyvanyk, Denys·2018·Proceedings of the 15th International Conference on Mining Software Repositories
DOI 10.1145/3196398.3196431
Open this paper in CoChat — free accountOpen PDF

Abstract

Assessing the similarity between code components plays a pivotal role in a number of Software Engineering (SE) tasks, such as clone detection, impact analysis, refactoring, etc. Code similarity is generally measured by relying on manually defined or hand-crafted features, e.g., by analyzing the overlap among identifiers or comparing the Abstract Syntax Trees of two code components. These features represent a best guess at what SE researchers can utilize to exploit and reliably assess code similarity for a given task. Recent work has shown, when using a stream of identifiers to represent the code, that Deep Learning (DL) can effectively replace manual feature engineering for the task of clone detection. However, source code can be represented at different levels of abstraction: identifiers, Abstract Syntax Trees, Control Flow Graphs, and Bytecode. We conjecture that each code representation can provide a different, yet orthogonal view of the same code fragment, thus, enabling a more reliable detection of similarities in code. In this paper, we demonstrate how SE tasks can benefit from a DL-based approach, which can automatically learn code similarities from different representations.

Integrity signals

Registry

Crossref · 10.1145/3196398.3196431

Cites

84 of 84 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 9 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    The combined model CloneDetector, trained on clone candidates from a single project (hibernate), can be effectively used to detect clones in the remaining nine systems.

    “We also show that the combined model CloneDetector, trained on clone candidates belonging to a single project (hibernate), can be effectively used to detect clones in the remaining nine systems.”
    Quote in the paperSubstantial claim
  • Findingown data

    Distances computed by the Identifier-based model for clones missed by it were several orders of magnitude higher than a reasonable threshold.

    “We found that the distances computed by the Identifier-based model for such clones were several orders of magnitude higher than a reasonable threshold”
    Quote in the paperSubstantial claim
  • Findingown data

    AST-based models identify the highest number of Type II clones.

    “Identifier-based models detect the highest number of Type I, AST-based models identify the highest number of Type II, Bytecode models identify the highest number of Type III, and CFG models detect the highest number of Type IV.”
    Quote in the paperSubstantial claim
  • Findingreasoning

    At class level, false positives for Bytecode and CFG are mainly due to Java Beans.

    “Better results are achieved at class-level, where false positives are mainly due to Java Beans”
    Quote in the paperNot a substantial claim
  • Findingown data

    The combined model achieves overall better results than stand-alone code representations.

    “In fact, we show that the combined model achieves overall better results as compared to stand-alone code representations.”
    Quote in the paperSubstantial claim
  • May combine two claims. 30
  • Classified as probably not a substantial claim. 44
  • Borderline: one of the tests for a substantial claim is unclear. 1
  • The quote only partly states this claim. 1

  • May combine two claims.

    Findingown data

    Most candidate clone partitions exhibit good precision, with peaks of 100%.

    “Table 2 shows that most of the partitions exhibit a good precision, with peaks of 100%.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The identifiers-based model has estimated recall of 52% at method level and 40% at class level.

    “However its estimated recall is respectively 52% and 40%, meaning that the model fails to detect a significant percentage of clones.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Clones missed by the Identifier-based model share a small percentage of identifiers, making them hard to detect with that representation.

    “a close inspection at the vocabulary of the candidates showed that they share a small percentage of identifiers, which in turn, makes them hard to detect with such a representation.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The proposed models can effectively compute code similarities both within a single project and among different software projects.

    “We also demonstrate that the proposed models can be effectively used to compute similarities, not only in the context of a single project, but also to analyze code similarities among different (diverse) software projects (e.g., detecting clones or libraries across multiple projects).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The precision for partitions ID 1 and 2 is 5% and 9% at method level and 49% and 58% at class level.

    “The precision for such partitions is 5% and 9% at method-level and 49% and 58% at class-level.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingnone given

    Developers took needed functionalities from Gamma, stripped unnecessary code, and built InternalGamma.

    “The developers only took the needed functionalities out of the Gamma class, stripped it of unnecessary code, and built InternalGamma.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    At class level, the CloneClassifier's overall F-measure was 68%, with average precision 67% and recall 68%.

    “At class level, the CloneClassifier has overall F-Measure of 68%, with average precision of 67% and recall of 68%.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingnone given

    The classes mentioned in the study refer to actual Java duplicated code, not imported libraries; the source files are in both libraries.

    “Note that the classes mentioned in this study do not refer to imported libraries but to actual Java duplicated code, i.e., the source code files are in both libraries.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning

    MessageDestination and ContextEjb share only a few identifiers and differ in some if-structures, making them more difficult to detect by CFG and Bytecode.

    “These classes share only a few identifiers (therefore not detected by the Identifier model) and differ in some if-structures, making them more difficult to detect by CFG and Bytecode representations.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The CloneDetector achieves 90% precision and 93% recall at class level, and 98% precision and 97% recall at method level.

    “When identifying clone instances, the model achieves 90% precision and 93% recall at class, and 98% precision 97% recall at method level.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Both classes implement a method that sets a seed.

    “As an example, both classes implement a method which sets a seed.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The classes ISAACRandom and Gamma/InternalGamma share many similarities across math3-3.6.1 and rng-1.0 but differ at statement and structural level.

    “The classes share many similarities across the two libraries, however, they differ on a statement and structural level.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    In one class, the seed is set by an integer argument; in the other, it is set using a combination of current time and system hash code.

    “In one class, the seed is set by an integer passed through an argument, while the other class sets the seed by using a combination of the current time and system hash code of the instance.”
    Quote in the paperNot a substantial claim

  • Borderline on S1 (asserted), S3 (load-bearing), S4 (based): unclear from the text.

    Findingnone given

    The strongest correlation observed is between AST and CFG, at 0.157.

    “the”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning

    Gamma and InternalGamma are clones because parts of Gamma were used to develop InternalGamma.

    “Gamma and InternalGamma, named respectively to the library they belong to, are clones since parts of the Gamma class were used to develop InternalGamma.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning

    EditScript and ReplacementsFinder classes are an example of a Type II clone within text-1.1 and collections4-4.1.

    “An example of Type II clone within these two libraries is instead the pair EditScript and ReplacementsFinder classes.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    These clones are classified as Type II because only package information changed.

    “These clones are classified as Type II, since only package information has been changed.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    At method level, the CloneClassifier's overall F-measure was 84%, and its average precision and recall were also 84% across categories.

    “At method level, the model obtains an overall F-Measure of 84%, with an average precision and recall of 84% across the different categories”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning

    Many methods in Gamma either do not appear or are significantly smaller in InternalGamma.

    “Therefore, many of the methods in Gamma either do not appear or are significantly smaller in InternalGamma.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given

    MessageDestination and ContextEjb both extend ResourceBase and offer the same features.

    “They both extend ResourceBase and offer the same features.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given

    Single-representation models obtain significantly worse results in the classification task, with a bigger gap compared to the CloneDetector.

    “Single-representation models obtain significantly worse results in the classification task (with a bigger gap with respect to what observed for the CloneDetector).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Other clone types achieve precision of at least 74% and recall of at least 75%.

    “while other clone types achieve a precision ≥ 74% and a recall ≥ 75%.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The CloneDetector achieved 98% precision and 92% recall at method level and 99% precision and 95% recall at class level.

    “It achieved 98% precision and 92% recall at method-level and 99% precision and 95% recall at class-level.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Premisenone given

    A language model and recursive autoencoder trained on a given vocabulary would not provide adequate results on a vocabulary with unseen terms.

    “For example, a language model and a recursive autoencoder trained on a given vocabulary would not be able to provide adequate results on a vocabulary containing terms not previously seen during the training phase.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisereasoning

    Learning transformations of data drastically reduces the cost of modeling code.

    “Firstly, learning transformations of the data drastically reduces the cost of modeling code, since software (and various code representations) store a lot of data to improve the effectiveness of learning.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The data presented in the results section is based on the agreement of three evaluators.

    “The data presented in this section is based on the agreement of three evaluators.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Premisenone given

    Bytecode and CFG representations have a very high degree of abstraction.

    “Bytecode and CFG representations have a very high degree of abstraction.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Java Beans are false positives in terms of code clones but represent successful classification in terms of learning structural and low-level similarities.

    “While these classes are false positives in terms of code clones, they still represent a successful classification in terms of learning structural and low-level similarities from the code.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data

    For RQ4, the authors verified whether identified classes were imported by downloading source code and analyzing Maven .pom files.

    “we downloaded the j's source code and analyzed the building files (weaver-1.3relies on Maven, thus we investigated the .pomfiles) to check whether the library x is imported as a dependency in j.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    Code similarity is generally measured using manually defined or hand-crafted features.

    “Code similarity is generally measured by relying on manually defined or hand-crafted features, e.g., by analyzing the overlap among identifiers or comparing the Abstract Syntax Trees of two code components.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    HOPE embeds directed graphs, which is needed for control flow graphs.

    “differently from other techniques (e.g., SDNE), HOPE embeds directed graphs (as needed in the case of Control Flow Graphs).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [55] 55. Annibale Panichella, Bogdan Dit, Rocco Oliveto, Massimiliano Di Penta, Denys Poshyvanyk, and Andrea De Lucia. 2016. Parameterizing and Assembling IR- Based Solutions for SE Tasks Using Genetic Algorithms. In IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering, SANER 2016, Suita, Osaka, Japan, March 14-18, 2016 -Volume 1. 314-325. https://doi.org/10.1109/ SANER.2016.97 source

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    Clone pairs with very high AST similarity but low identifier similarity are likely Type-II clones.

    “For example, clone pairs with very high AST similarity, but low identifiers similarity are likely to be Type-II clones (possibly parameterized clones, where all the identifiers have been systematically renamed).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation

    Fleiss' kappa was computed to statistically assess evaluators' agreement.

    “In order to statistically assess the evaluators' agreement, we compute the Fleiss' kappa [24].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [24] 24. M. Gabel, L. Jiang, and Z. Su. [n. d.]. Scalable Detection of Semantic Clones (ICSE'08).

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Study design and conductreasoning

    Using the Qualitas.classCorpus dataset avoids potential compilation problems or inconsistencies and ensures reproducibility.

    “This (i) avoids any potential problem/inconsistency in compiling projects, and (ii) ensures reproducibility.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    Similarities from different code representations can be useful not only to detect clones, but also to classify them into types.

    “Similarities from different code representations can be useful not only to detect clones, but also to classify them into types.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given

    If bytecode is available, CFGs can be extracted as well.

    “Indeed, if bytecode is available (compiled code) CFGs can be extracted as well.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisenone given

    AST and Bytecode representations both have a fixed vocabulary, limited by the number of AST node types and bytecode opcodes, respectively.

    “In our study, AST and Bytecode representations both have a fixed vocabulary, limited respectively by the number of different AST node types and bytecode opcodes.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    No other learning-based method is known to use representations beyond identifiers and comments.

    “However, we are not aware of any other learning-based approach that operates on code representations other than identifiers and comments.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    The candidate clone sample was drawn with 95% confidence level and ±15% confidence interval.

    “we randomly draw a statistically significant sample with 95% confidence level and ±15% confidence interval.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    The two combined models were trained using Random Forest with 10-fold cross-validation.

    “To train the two models we rely on Random Forest employing the commonly used 10-fold cross-validation technique to partition training and test sets.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    Using the same thresholds across representations facilitates comparison among the four representations.

    “While, ideally, these thresholds should be tuned for each project and representation, we chose the same thresholds to facilitate the comparison among the four representations.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Random Forest is an effective ensemble learning method for classification, relying on decision tree learning and bagging.

    “Random Forest is one of the most effective ensemble learning method for classification, which relies on decision tree learning and bagging [17].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [17] 17. Bogdan Dit, Latifa Guerrouj, Denys Poshyvanyk, and Giuliano Antoniol. 2011. Can Better Identifier Splitting Techniques Help Feature Location?. In Proceedings of the 2011 IEEE 19th International Conference on Program Comprehension (ICPC '11). IEEE Computer Society, Washington, DC, USA, 11-20. https://doi.org/10. 1109/ICPC.2011.47 source

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationnone given

    The paper's results should be useful for any software engineering task that relies on analyzing code similarities.

    “We argue that our results should be useful for any SE task that relies on analyzing code similarities.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    Deep learning can effectively replace manual feature engineering for clone detection when code is represented as a stream of identifiers.

    “Recent work has shown, when using a stream of identifiers to represent the code, that Deep Learning (DL) can effectively replace manual feature engineering for the task of clone detection.”
    • Also stated in: § 1 INTRODUCTION · plain
      “one recent example by White et al. [74] shows that DL can effectively replace manual feature engineering for the task of clone detection.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites White et al. [74]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    Learning algorithms typically outperform humans at finding correlations in high-dimensional spaces.

    “Secondly, performance is increased because generally, learning algorithms are more efficient than humans at discovering correlations in high dimensional spaces.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    Three authors independently evaluated the clone candidate samples.

    “Three authors independently evaluated the clone candidate samples.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    The final dataset was obtained by two-judges agreement, with a candidate marked true positive if at least two of three evaluators classified it as such.

    “we computed the two-judges agreement to obtain the final dataset (e.g., a candidate clone was marked as a true positive if at least two of the three evaluators classified it as such).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationown data

    The lower performance at class level could be due to the smaller dataset available (483 methods vs 362 classes).

    “The lower performance at class-level could be due to the smaller dataset available (i.e., 483 methods vs 362 classes).”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Interpretationown data

    The small overlap among candidate sets suggests that the representations complement each other.

    “suggests that these representations complement each other.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The number of instances and percentages for exclusive candidates are computed only on the manually validated sample.

    “Note that the number of instances and percentages are computed only on the manually validated sample.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    The low correlation values suggest that distances computed with different representations do not correlate and provide different perspectives about code pair similarity.

    “their low value suggest that distances computed with different representations do not correlate and provide different perspectives about the similarity of code pairs.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    This allows SE researchers to make a more targeted choice of code representation when applying representation-DL algorithms to SE tasks.

    “This allows for a more targeted choice by SE researchers of a code representation when applying representation-DL algorithms to a SE tasks.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    The approach was not compared against code clone detection techniques because the focus is on a general technique rather than building a code clone detector tool.

    “We do not compare our approach against code clone detection techniques since the focus of this paper is to show a general technique on how to learn similarities from multiple code representations, rather than building a code clone detector tool.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductreasoning

    To mitigate the external validity threat, the approach was applied in different contexts and two datasets, Projects and Libraries, were used.

    “To mitigate this threat, we applied our approach in different contexts and used two different datasets; Projects and Libraries.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    All Libraries, including Apache commons libraries, are publicly available, ensuring replicability.

    “All the Libraries, including all the Apache commons libraries, are publicly available, ensuring the replicability of the study.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    All results are available in the online appendix.

    “All the results are available in our online appendix.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Interpretationreasoning

    These findings speak to the vast amount of SE tasks that benefit from analyzing multiple representations of code.

    “These findings speak to the vast amount of SE tasks which benefit from analyzing multiple representations of the code.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    This work emphasizes attributes that different code representations can accentuate.

    “This work also emphasizes the attributes that different code representations can accentuate.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation

    All evaluations are publicly available.

    “Also, all our evaluations are publicly available [2].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [2] 2. M. Allamanis, E. Barr, C. Bird, and C. Sutton. [n. d.]. Suggesting Accurate Method and Class Names (FSE'15).

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    Two-judge agreement was computed on the evaluated candidates.

    “Then, we computed two-judge agreement on the evaluated candidates.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    The results using the selected datasets might not generalize to other projects.

    “The results obtained in our study using the selected datasets might not generalize to other projects.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Representation extraction steps are implemented for Java, but all subsequent steps are language-agnostic because they rely on a corpus of sentences with arbitrary tokens.

    “While the representation extraction steps are implemented for Java, all the subsequent steps are completely language-agnostic because they rely on a corpus of sentences with arbitrary tokens.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Recursive Autoencoders have previously been applied in multiple domains and with various input types, including natural language, source code, and images.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    Other clone-focused datasets such as BigCloneBench were not used because extracting CFG and Bytecode required compiled code.

    “We did not utilize other clone-focused datasets (e.g., such as BigCloneBench [66]) since in order to extract some representations (i.e., CFG and Bytecode) we needed compiled code.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [66] 66. Ricardo Terra, Luis Fernando Miranda, Marco Tulio Valente, and Roberto S. Bigonha. 2013. Qualitas.class Corpus: A Compiled Version of the Qualitas Corpus. Software Engineering Notes 38, 5 (2013), 1-4. source

  • Classified as probably not a substantial claim.

    Study design and conductown data

    The complementarity claim is supported by complementarity metrics and a correlation test.

    “We support this claim by performing different analyses: (i) complementarity metrics; and (ii) correlation test.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    Three evaluators independently checked the candidates.

    “To mitigate such threat, we employed three evaluators who independently checked the candidates.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    For Projects, a subset of the Qualitas.class corpus, diverse systems in size and domain were selected, focusing on popular ones.

    “For Projects, which is a subset of systems from the Qualitas.classcorpus, we selected diverse systems in terms of size and domain, focusing on popular ones.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Reusability eliminates a large timesink native to DL approaches and broadens applicability.

    “This eliminates a large timesink, native to DL approaches, and broadens the applicability of our approach.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The approach was instantiated in different use case scenarios and datasets.

    “We instantiated our approach in different use case scenarios and datasets.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    The approach was applied only to Java code.

    “Another threat in this category is related to the fact that we apply our approach on Java code only.”
    Quote in the paperNot a substantial claim
69 of 69

  • The paper relies on this without support.

    Deep learning models require a long time to train.

    “One of the drawbacks of DL-based models is their long training time.”

  • The paper relies on this without support.

    Enumerators tend to have bytecode structures that are highly similar even when their constants differ.

    “Enumerators have a very similar bytecode structure even if containing different constants.”

  • The paper relies on this without support.

    Recursive Autoencoders have previously been applied in multiple domains and with various input types, including natural language, source code, and images.

    “As a matter of fact, Recursive Autoencoders have been used in several contexts with different inputs such as natural language, source code, images.”
3 of 3

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Class-level Type III results contradict the stated minimums

    Minor

    The text says every clone type other than Type IV has precision of at least 74% and recall of at least 75%. The table gives class-level Type III clones 61% precision and 59% recall.

  • Clone-detection citation identifies the wrong author

    Minor

    The text attributes the clone-detection work to White et al. at reference 74, but reference 74 is an entry by Zhou on ensemble learning.

  • Class-level Type III results fall below the claimed minimums

    Minor

    The paper says every clone type other than Type IV reaches at least 74% precision and 75% recall. Table 7 reports only 61% precision and 59% recall for class-level Type III clones.

  • Data-release citation points to a different work

    Minor

    The citation attached to the data-release claim points to a reference about suggesting method and class names, not to the paper's released data.

4 of 4

Cites

Flagged first, then by citations

Research works printed but not linked: 9

  1. 2017. Online Appendix: https://sites.google.com/view/learningcodesimilarities , Source Code: https://github.com/micheletufano/AutoenCODE. ( 2017 ). 2017. Online Appendix: https://sites.google.com/view/learningcodesimilarities, Source Code: https://github.com/micheletufano/AutoenCODE. (2017).
  2. 10.5555/2486788.2486879 · DOI
  3. 10.5555/3060832.3060845 · DOI
  4. 10.5555/2337223.2337267 · DOI
  5. 10.5555/3015812.3016002 · DOI
  6. 10.5555/2486788.2486857 · DOI
  7. 10.5555/2662708.2662719 · DOI
  8. 10.1145/2491411.2491420 · DOI · Paper page
  9. 10.5555/3172077.3172312 · DOI

Cited by

Sorted by citations

Related

Authors
Tufano, M., Watson, C., Bavota, G., Di Penta, M., White, M., & Poshyvanyk, D.
Affiliations
  • William & Mary (US)
  • Università della Svizzera italiana (CH)
  • University of Sannio, Benevento, Italy
ORCID
Venue
Proceedings of the 15th International Conference on Mining Software Repositories
Publisher
ACM
Published
2018-05-28
Pages
542-553
DOI
10.1145/3196398.3196431
OpenAlex
W2883359218
Cited by
158
References
75
Citations by year
2026: 6 · 2025: 11 · 2024: 20 · 2023: 22 · 2022: 31 · 2021: 29 · 2020: 20 · 2019: 18 · 2018: 1
FWCI
13.34
Type
conference paper
Language
en
Open access
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor)
Locations
International Conference on Mining Software Repositories (MSR) (publishedVersion)
Funders
NSF (CCF-1525902)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
Keywords
code similarity, clone detection, source code representation, abstract syntax tree, control flow graph, bytecode, feature engineering

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read under definition version 3 (extract-v3) on 2026-09-28.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-09-28.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review