Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Sign Up
Cited by 622

The DOI matches its Crossref record; 10 of the 10 references it cites were found in a registry or an index.

A Hitchhiker's guide to statistical tests for assessing randomized algorithms in software engineering

Open access
Arcuri, Andrea, Briand, Lionel·2014·Software Testing, Verification and Reliability
DOI 10.1002/stvr.1486
Open this paper in CoChat — free account

Abstract

Randomized algorithms are widely used to address many types of software engineering problems, especially in the area of software verification and validation with a strong emphasis on test automation. However, randomized algorithms are affected by chance and so require the use of appropriate statistical tests to be properly analysed in a sound manner. This paper features a systematic review regarding recent publications in 2009 and 2010 showing that, overall, empirical analyses involving randomized algorithms in software engineering tend to not properly account for the random nature of these algorithms. Many of the novel techniques presented clearly appear promising, but the lack of soundness in their empirical evaluations casts unfortunate doubts on their actual usefulness. In software engineering, although there are guidelines on how to carry out empirical analyses involving human subjects, those guidelines are not directly and fully applicable to randomized algorithms. Furthermore, many of the textbooks on statistical analysis are written from the viewpoints of social and natural sciences, which present different challenges from randomized algorithms. To address the questionable overall quality of the empirical analyses reported in the systematic review, this paper provides guidelines on how to carry out and properly analyse randomized algorithms applied to solve software engineering tasks, with a particular focus on software testing, which is by far the most frequent application area of randomized algorithms within software engineering. Copyright © 2012 John Wiley & Sons, Ltd.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1002/stvr.1486

Cites

10 of 10 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 10 references

Research works printed but not linked: 117

  1. Harman, 2010, A theoretical and empirical study of search based testing: local, global and hybrid search, IEEE Transactions on Software Engineering (TSE), 36, 226, 10.1109/TSE.2009.71 · DOI · Paper page
  2. Godefroid P Klarlund N Sen K DART: directed automated random testing ACM Conference on Programming Language Design and Implementation (PLDI) 2005 213 223 · DOI · Paper page
  3. Motwani, 1995, Randomized Algorithms, 10.1017/CBO9780511814075 · DOI
  4. Arcuri, 2012, Random testing: theoretical results and practical implications, IEEE Transactions on Software Engineering (TSE), 38, 258, 10.1109/TSE.2011.121 · DOI
  5. Duran, 1984, An evaluation of random testing, IEEE Transactions on Software Engineering (TSE), 10, 438, 10.1109/TSE.1984.5010257 · DOI
  6. Arcuri A Briand L A practical guide for using statistical tests to assess randomized algorithms in software engineering ACM/IEEE International Conference on Software Engineering (ICSE) 2011 1 10 · DOI · Paper page
  7. Harman M Mansouri SA Zhang Y 2009 Search based software engineering: a comprehensive analysis and review of trends techniques and applications Technical Report TR-09-03
  8. McMinn, 2004, Search-based software test data generation: a survey, Software Testing, Verification and Reliability, 14, 105, 10.1002/stvr.294 · DOI · Paper page
  9. Bagnall, 2001, The next release problem, Information and Software Technology, 43, 883, 10.1016/S0950-5849(01)00194-X · DOI · Paper page
  10. Aguilar-Ruiz, 2001, An evolutionary approach to estimating software development projects, Information and Software Technology, 43, 875, 10.1016/S0950-5849(01)00193-8 · DOI
  11. Arcuri A Yao X A novel co-evolutionary approach to automatic software bug fixing IEEE Congress on Evolutionary Computation (CEC) 2008 162 168 · DOI
  12. Mitchell, 2006, On the automatic modularization of software systems using the bunch tool, IEEE Transactions on Software Engineering (TSE), 32, 193, 10.1109/TSE.2006.31 · DOI · Paper page
  13. Canfora G Penta MD Esposito R Villani ML An approach for QoS-aware service composition based on genetic algorithms Genetic and Evolutionary Computation Conference (GECCO) 2005 1069 1075 · DOI · Paper page
  14. Cooper KD Schielke PJ Subramanian D Optimizing for reduced code space using genetic algorithms Proceedings of the ACM SIGPLAN Workshop on Languages, Compilers, and Tools for Embedded Systems 1999 1 9 · DOI
  15. Khoshgoftaar, 2004, A multiobjective module-order model for software quality enhancement, IEEE Transactions on Evolutionary Computation (TEC), 8, 593, 10.1109/TEVC.2004.837108 · DOI
  16. Grissom, 2005, Effect Sizes for Research: A Broad Practical Approach
  17. Klein, 2003, Survival Analysis: Techniques for Censored and Truncated Data, 10.1007/b97377 · DOI · Paper page
  18. Rice, 1994, Mathematical Statistics and Data Analysis
  19. Dybå, 2006, A systematic review of statistical power in software engineering experiments, Information and Software Technology (IST), 48, 745, 10.1016/j.infsof.2005.08.009 · DOI · Paper page
  20. Kampenes, 2007, A systematic review of effect size in software engineering experiments, Information and Software Technology (IST), 49, 1073, 10.1016/j.infsof.2007.02.015 · DOI · Paper page
  21. Nakagawa, 2007, Effect size, confidence interval and statistical significance: a practical guide for biologists, Biological Reviews, 82, 591, 10.1111/j.1469-185X.2007.00027.x · DOI · Paper page
  22. Katz, 2006, Multivariable Analysis: A Practical Guide for Clinicians, 10.1017/CBO9780511811692 · DOI · Paper page
  23. Wohlin, 2000, Experimentation in Software Engineering: An Introduction, 6, 10.1007/978-1-4615-4625-2 · DOI
  24. Poulding, Efficient software verification: statistical testing using automated search, IEEE Transactions on Software Engineering (TSE), 36, 763, 10.1109/TSE.2010.24 · DOI
  25. Ali, 2010, A systematic review of the application and empirical investigation of search-based test-case generation, IEEE Transactions on Software Engineering (TSE), 36, 742, 10.1109/TSE.2009.52 · DOI · Paper page
  26. Feller, 1968, An Introduction to Probability Theory and Its Applications, 1
  27. Khan, 2004, Systematic Reviews to Support Evidence-Based Medicine: How to Review and Apply Findings of Healthcare Research
  28. Hsu H Orso A MINTS: a general framework and tool for supporting test-suite minimization ACM/IEEE International Conference on Software Engineering (ICSE) 2009 419 429 · DOI
  29. Thum T Batory D Kastner C Reasoning about edits to feature models ACM/IEEE International Conference on Software Engineering (ICSE) 2009 254 264 · DOI
  30. Mitchell, 1997, Machine Learning
  31. Ganesh V Leek T Rinard M Taint-based directed whitebox fuzzing ACM/IEEE International Conference on Software Engineering (ICSE) 2009 474 484 · DOI · Paper page
  32. Abraham, 2009, Mutation operators for spreadsheets, IEEE Transactions on Software Engineering (TSE), 35, 94, 10.1109/TSE.2008.73 · DOI
  33. Masood, 2009, Scalable and effective test generation for role-based access control systems, IEEE Transactions on Software Engineering (TSE), 35, 654, 10.1109/TSE.2009.35 · DOI
  34. Ngo-The, 2009, Optimized resource allocation for software release planning, IEEE Transactions on Software Engineering (TSE), 35, 109, 10.1109/TSE.2008.80 · DOI
  35. Menzies T Williams S Boehm B Hihn J How to avoid drastic software process change (using stochastic stability) ACM/IEEE International Conference on Software Engineering (ICSE) 2009 540 550 · DOI
  36. Weimer W Nguyen T Goues CL Forrest S Automatically finding patches using genetic programming ACM/IEEE International Conference on Software Engineering (ICSE) 2009 364 374 · DOI · Paper page
  37. Kieyzun A Guo P Jayaraman K Ernst M Automatic creation of SQL injection and cross-site scripting attacks ACM/IEEE International Conference on Software Engineering (ICSE) 2009 199 209 · DOI · Paper page
  38. Arcuri A Full theoretical runtime analysis of alternating variable method on the triangle classification problem International Symposium on Search Based Software Engineering (SSBSE) 2009 113 121 · DOI
  39. Ghani K Clark J Heslington Y Widening the goal posts: program stretching to aid search based software testing International Symposium on Search Based Software Engineering (SSBSE) 2009 122 131 · DOI
  40. Durillo J Zhang Y Alba E Nebro A A study of the multi-objective next release problem International Symposium on Search Based Software Engineering (SSBSE) 2009 49 58 · DOI
  41. Garvin B Cohen M Dwyer M An improved meta-heuristic search for constrained interaction testing International Symposium on Search Based Software Engineering (SSBSE) 2009 13 22 · DOI
  42. Kpodjedo S Ricca F Antoniol G Galinier P Evolution and search based metrics to improve defects prediction International Symposium on Search Based Software Engineering (SSBSE) 2009 23 32 · DOI
  43. Khan U Bate I WCET analysis of modern processors using multi-criteria optimisation International Symposium on Search Based Software Engineering (SSBSE) 2009 103 112 · DOI
  44. Marchetto A Tonella P Search-based testing of Ajax web applications International Symposium on Search Based Software Engineering (SSBSE) 2009 3 12 · DOI
  45. Kim D Park S Dynamic architectural selection: a genetic algorithm based approach International Symposium on Search Based Software Engineering (SSBSE) 2009 59 68 · DOI
  46. Shevertalov M Kothari J Stehle E Mancoridis S On the use of discretized source code metrics for author identification International Symposium on Search Based Software Engineering (SSBSE) 2009 69 78 · DOI
  47. Bryce, 2009, A density-based greedy algorithm for higher strength covering arrays, Software Testing, Verification and Reliability (STVR), 19, 37, 10.1002/stvr.393 · DOI
  48. Polo, 2009, Decreasing the cost of mutation testing with second-order mutants, Software Testing, Verification and Reliability (STVR), 19, 111, 10.1002/stvr.392 · DOI
  49. Schneidewind, 2009, Integrating testing with reliability, Software Testing, Verification and Reliability (STVR), 19, 175, 10.1002/stvr.395 · DOI
  50. Huo, 2009, Transition covering tests for systems with queues, Software Testing, Verification and Reliability (STVR), 19, 55, 10.1002/stvr.396 · DOI
  51. White, 2010, ASCENT: an algorithmic technique for designing hardware and software in tandem, IEEE Transactions on Software Engineering (TSE), 36, 10.1109/TSE.2010.77 · DOI
  52. Garousi, 2010, A genetic algorithm-based stress test requirements generator tool and its empirical evaluation, IEEE Transactions on Software Engineering (TSE), 36, 778, 10.1109/TSE.2010.5 · DOI
  53. Yuan, 2010, Generating event sequence-based test cases using GUI runtime state feedback, IEEE Transactions on Software Engineering (TSE), 36, 81, 10.1109/TSE.2009.68 · DOI
  54. Do, 2010, The effects of time constraints on test case prioritization: a series of controlled experiments, IEEE Transactions on Software Engineering (TSE), 36, 593, 10.1109/TSE.2010.58 · DOI
  55. Simons, 2010, Interactive, evolutionary search in upstream object-oriented class design, IEEE Transactions on Software Engineering (TSE), 36, 798, 10.1109/TSE.2010.34 · DOI
  56. Bowman, 2010, Solving the class responsibility assignment problem in object-oriented analysis with multi-objective genetic algorithms, IEEE Transactions on Software Engineering (TSE), 36, 817, 10.1109/TSE.2010.70 · DOI
  57. Emberson, 2010, Stressing search with scenarios for flexible solutions to real-time task allocation problems, IEEE Transactions on Software Engineering (TSE), 36, 704, 10.1109/TSE.2009.58 · DOI
  58. Antunes, 2010, Vulnerability discovery with attack injection, IEEE Transactions on Software Engineering (TSE), 36, 357, 10.1109/TSE.2009.91 · DOI
  59. Artzi, 2010, Finding bugs in web applications using dynamic test generation and explicit-state model checking, IEEE Transactions on Software Engineering (TSE), 36, 474, 10.1109/TSE.2010.31 · DOI
  60. Beckman, 2010, Proofs from tests, IEEE Transactions on Software Engineering (TSE), 36, 495, 10.1109/TSE.2010.49 · DOI
  61. Lai Z Cheung S Chan W Detecting atomic-set serializability violations in multithreaded programs through active randomized testing ACM/IEEE International Conference on Software Engineering (ICSE) 2010 235 244 · DOI
  62. Zhang L Hou S Hu J Xie T Mei H Is operator-based mutant selection superior to random mutant selection? ACM/IEEE International Conference on Software Engineering (ICSE) 2010 435 444 · DOI
  63. Gligoric M Gvero T Jagannath V Khurshid S Kuncak V Marinov D Test generation through programming in UDITA ACM/IEEE International Conference on Software Engineering (ICSE) 2010 225 234 · DOI
  64. Nainar PA Liblit B Adaptive bug isolation ACM/IEEE International Conference on Software Engineering (ICSE) 2010 255 264 · DOI
  65. Gabel M Su Z Online inference and enforcement of temporal properties ACM/IEEE International Conference on Software Engineering (ICSE) 2010 15 24 · DOI
  66. Gu Z Barr ET Hamilton DJ Su Z Has the bug really been fixed? ACM/IEEE International Conference on Software Engineering (ICSE) 2010 55 64 · DOI
  67. Jha S Gulwani S Seshia SA Tiwari A Oracle-guided component-based program synthesis ACM/IEEE International Conference on Software Engineering (ICSE) 2010 215 224 · DOI · Paper page
  68. Yang Q Li M A cut-off approach for bounded verification of parameterized systems ACM/IEEE International Conference on Software Engineering (ICSE) 2010 345 354 · DOI
  69. Nori A Rajamani SK An empirical study of optimizations in yogi ACM/IEEE International Conference on Software Engineering (ICSE) 2010 355 364 · DOI
  70. Schaefer CA Pankratius V Tichy WF Engineering parallel applications with tunable architectures ACM/IEEE International Conference on Software Engineering (ICSE) 2010 405 414 · DOI
  71. de Souza JT Maia CL de Freitas FG Coutinho DP The human competitiveness of search based software engineering International Symposium on Search Based Software Engineering (SSBSE) 2010 143 152 · DOI
  72. del Sagrado J del Aguila IM Orellana FJ Ant colony optimization for the next release problem International Symposium on Search Based Software Engineering (SSBSE) 2010 67 76 · DOI
  73. Lu G Bahsoon R Yao X Applying elementary landscape analysis to search-based software engineering International Symposium on Search Based Software Engineering (SSBSE) 2010 3 8 · DOI
  74. McMinn P How does program structure impact the effectiveness of the crossover operator in evolutionary testing? International Symposium on Search Based Software Engineering (SSBSE) 2010 9 18 · DOI
  75. Xiao J Afzal W Search-based resource scheduling for bug fixing tasks International Symposium on Search Based Software Engineering (SSBSE) 2010 133 142 · DOI
  76. Yoo S A novel mask-coding representation for set cover problems with applications in test suite minimisation International Symposium on Search Based Software Engineering (SSBSE) 2010 19 28 · DOI
  77. Lakhotia K Harman M Gross H AUSTIN: a tool for search based software testing for the C language and its evaluation on deployed automotive systems International Symposium on Search Based Software Engineering (SSBSE) 2010 101 110 · DOI
  78. Tonella P Susi A Palma F Using interactive GA for requirements prioritization International Symposium on Search Based Software Engineering (SSBSE) 2010 57 66 · DOI
  79. Asadi F Antoniol G Gueheneuc Y Concept location with genetic algorithms: a comparison of four distributed architectures International Symposium on Search Based Software Engineering (SSBSE) 2010 153 162 · DOI
  80. Lindlar F Windisch A A search-based approach to functional hardware-in-the-loop testing International Symposium on Search Based Software Engineering (SSBSE) 2010 111 119 · DOI
  81. Zhang Y Harman M Search based optimization of requirements interaction management International Symposium on Search Based Software Engineering (SSBSE) 2010 47 56 · DOI
  82. Zhao, 2010, Automatic string test data generation for detecting domain errors, Software Testing, Verification and Reliability (STVR), 20, 209, 10.1002/stvr.414 · DOI
  83. Griesmayer, 2006, Repair of Boolean programs with an application to C, Computer Aided Verification, 358, 10.1007/11817963_33 · DOI
  84. Staber S Jobstmann B Bloem R Finding and fixing faults Conference on Correct Hardware Design and Verification Methods (CHARME) 2005 35 49 · DOI
  85. Stumptner M Wotawa F A model based approach to software debugging International Workshop on Principles of Diagnosis 1996
  86. Cowles, 1982, On the origins of the .05 level of statistical significance, American Psychologist, 37, 553, 10.1037/0003-066X.37.5.553 · DOI · Paper page
  87. Goodman, 1993, P values, hypothesis tests, and likelihood: implications for epidemiology of a neglected historical debate, American Journal of Epidemiology, 137, 485, 10.1093/oxfordjournals.aje.a116700 · DOI · Paper page
  88. Goodman, 1999, Toward evidence-based medical statistics. 1: the P value fallacy, Annals of Internal Medicine, 130, 995, 10.7326/0003-4819-130-12-199906150-00008 · DOI · Paper page
  89. Arcuri, 2012, Formal analysis of the probability of interaction fault detection using random testing, IEEE Transactions on Software Engineering (TSE), 38, 1088, 10.1109/TSE.2011.85 · DOI
  90. Sharma R Gligoric M Arcuri A Fraser G Marinov D Testing container classes: random or systematic? Fundamental Approaches to Software Engineering (FASE) 2011 262 277 · DOI
  91. Fraser G Arcuri A Evolutionary generation of whole test suites International Conference on Quality Software (QSIC) 2011 31 40 · DOI
  92. Siegmund, 1985, Sequential Analysis: Tests and Confidence Intervals, 10.1007/978-1-4757-1862-1 · DOI · Paper page
  93. Fay, 2010, Wilcoxon-Mann-Whitney or t-test? On assumptions for hypothesis tests and multiple interpretations of decision rules, Statistics Surveys, 4, 1, 10.1214/09-SS051 · DOI · Paper page
  94. Sawilowsky, 1992, A more realistic look at the robustness and type II error properties of the t test to departures from population normality, Psychological Bulletin, 111, 352, 10.1037/0033-2909.111.2.352 · DOI · Paper page
  95. Nijssen, 2003, An analysis of the behavior of simplified evolutionary algorithms on trap functions, IEEE Transactions on Evolutionary Computation (TEC), 7, 11, 10.1109/TEVC.2002.806169 · DOI
  96. Rudolph, 1994, Convergence analysis of canonical genetic algorithms, IEEE Transactions on Neural Networks, 5, 96, 10.1109/72.265964 · DOI · Paper page
  97. Tonella P Evolutionary testing of classes ACM International Symposium on Software Testing and Analysis (ISSTA) 2004 119 128 · DOI
  98. Freitag, 2006, Non-parametric assessment of non-inferiority with censored data, Statistics in Medicine, 25, 1201, 10.1002/sim.2444 · DOI
  99. Arcuri A Theoretical analysis of local search in software testing Symposium on Stochastic Algorithms, Foundations and Applications (SAGA) 2009 156 168 · DOI
  100. Vargha, 2000, A critique and improvement of the CL common language effect size statistics of McGraw and Wong, Journal of Educational and Behavioral Statistics, 25, 101
  101. Chernick M Bootstrap Methods: A Practitioner's Guide (Wiley Series in Probability and Statistics 1999
  102. Arcuri A Iqbal MZ Briand L Black-box system testing of real-time embedded systems using random and search-based testing IFIP International Conference on Testing Software and Systems (ICTSS) 2010 95 110 · DOI
  103. Arcuri A Fraser G On parameter tuning in search based software engineering SSBSE 2011 33 47 · DOI
  104. Nakagawa, 2004, A farewell to Bonferroni: the problems of low statistical power and publication bias, Behavioral Ecology, 15, 1044, 10.1093/beheco/arh107 · DOI · Paper page
  105. García, 2004, Escaping the Bonferroni iron claw in ecological studies, Oikos, 105, 657, 10.1111/j.0030-1299.2004.13046.x · DOI · Paper page
  106. Carrano, 2011, A multicriteria statistical based comparison methodology for evaluating evolutionary algorithms, IEEE Transactions on Evolutionary Computation (TEC), 15, 848, 10.1109/TEVC.2010.2069567 · DOI
  107. Fraser G Arcuri A It is not the length that matters, it is how you control it IEEE International Conference on Software Testing, Verification and Validation (ICST) 2011 150 159 · DOI
  108. Tillmann N de Halleux NJ Pex-white box test generation for .NET International Conference on Tests And Proofs (TAP) 2008 134 253 · DOI · Paper page
  109. Pacheco C Lahiri SK Ernst MD Ball T Feedback-directed random test generation ACM/IEEE International Conference on Software Engineering (ICSE) 2007 75 84 · DOI · Paper page
  110. Andrews, 2011, Genetic algorithms for randomized unit testing, IEEE Transactions on Software Engineering (TSE), 37, 80, 10.1109/TSE.2010.46 · DOI
  111. Arcuri, 2008, Search based software testing of object-oriented containers, Information Sciences, 178, 3075, 10.1016/j.ins.2007.11.024 · DOI
  112. Ribeiro, 2009, Test case evaluation and input domain reduction strategies for the evolutionary testing of object-oriented software, Information and Software Technology, 51, 1534, 10.1016/j.infsof.2009.06.009 · DOI
  113. Shousha, 2012, A UML/MARTE model analysis method for uncovering scenarios leading to starvation and deadlocks in concurrent systems, IEEE Transactions on Software Engineering (TSE), 38, 354, 10.1109/TSE.2010.107 · DOI
  114. Fraser G Arcuri A Sound empirical evidence in software testing ACM/IEEE International Conference on Software Engineering (ICSE) 2012 178 188 · DOI
  115. Alshraideh, 2006, Search-based software test data generation for string data using program-specific search operators, Software Testing, Verification and Reliability (STVR), 16, 175, 10.1002/stvr.354 · DOI
  116. Hemmati H Arcuri A Briand L Empirical investigation of the effects of test suite properties on similarity-based test case selection IEEE International Conference on Software Testing, Verification and Validation (ICST) 2011 327 336 · DOI
  117. R Development Core Team, 2008, R: A Language and Environment for Statistical Computing

Cited by

Metadata

Authors
Arcuri, A., & Briand, L.
Venue
Software Testing, Verification and Reliability
Publisher
Wiley
Published
2012-11-06
Volume
24
Issue
3
Pages
219-250
ISSN
0960-0833 (ISSN-L)
DOI
10.1002/stvr.1486
OpenAlex
W1526710119
Cited by
622
References
10
FWCI
58.87
Type
journal article
Language
en
Open access
green
Licence
doi.wiley.com/10.1002/tdm_license_1.1 (tdm, 1029 d)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.