A Survey of Field-based Testing Techniques
Open accessThe DOI matches its Crossref record; 107 of the 107 references it cites were found in a registry or an index.
Abstract
Field testing refers to testing techniques that operate in the field to reveal those faults that escape in-house testing. Field testing techniques are becoming increasingly popular with the growing complexity of contemporary software systems. In this article, we present the first systematic survey of field testing approaches over a body of 80 collected studies, and propose their categorization based on the environment and the system on which field testing is performed. We discuss four research questions addressing how software is tested in the field, what is tested in the field, which are the requirements , and how field tests are managed , and identify many challenging research directions.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The approaches to field testing monitor the applications in production to enrich the test suites with new test cases [16].”
From A 2030 Roadmap for Software Engineering (2025) · cites this paper as [16]
Citations
Cites
Lecture Notes in Computer Science 1205
(1999) · Industrial Robot: An International Journal · doi:10.1108/ir.1999.04926fae.001 · Cited by 38,662
Guidelines for conducting systematic mapping studies in software engineering: An update
Petersen, K., Vakkalanka, S., & Kuzniarz, L. (2015) · Information and Software Technology · doi:10.1016/j.infsof.2015.03.007 · Cited by 2,669
The state of the art in automating usability evaluation of user interfaces
Ivory, M. Y., & Hearst, M. A. (2001) · ACM Computing Surveys · doi:10.1145/503112.503114 · Cited by 875 · cited 1×
“• Exclusion Criterion 2: studies purely focusing on techniques other than testing, most notably we exclude works centered on: runtime verification [68], or debugging, even approaches for debugging in the field such as [65], or continuous experimentation via user surveys [90] and A/B testing [57], or usability evaluati…”
BackgroundA brief account of runtime verification
Leucker, M., & Schallhart, C. (2009) · The Journal of Logic and Algebraic Programming · doi:10.1016/j.jlap.2008.08.004 · Cited by 863 · cited 1×
“• Exclusion Criterion 2: studies purely focusing on techniques other than testing, most notably we exclude works centered on: runtime verification [68], or debugging, even approaches for debugging in the field such as [65], or continuous experimentation via user surveys [90] and A/B testing [57], or usability evaluati…”
BackgroundContinuous software engineering: A roadmap and agenda
Fitzgerald, B., & Stol, K.-J. (2017) · Journal of Systems and Software · doi:10.1016/j.jss.2015.06.063 · Cited by 723 · cited 1×
“Nowadays, the emergence of more and more agile and distributed paradigms of development, toward the view of a continuous software engineering discipline [35], emphasizes the need for a continuous testing mindset, making developers realize that testing must continue after deployment, and that no clear boundary can be s…”
BackgroundUser experience evaluation methods: current state and development needs
Vermeeren, A. P. O. S., Law, E. L.-C., Roto, V., Obrist, M., Hoonhout, J., & Väänänen-Vainio-Mattila, K. (2010) · Proceedings of the 6th Nordic Conference on Human-Computer Interaction: Extending Boundaries · doi:10.1145/1868914.1868973 · Cited by 592 · cited 1×
“• Exclusion Criterion 2: studies purely focusing on techniques other than testing, most notably we exclude works centered on: runtime verification [68], or debugging, even approaches for debugging in the field such as [65], or continuous experimentation via user surveys [90] and A/B testing [57], or usability evaluati…”
BackgroundA Survey on Metamorphic Testing
Segura, S., Fraser, G., Sanchez, A. B., & Ruiz-Cortes, A. (2016) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2016.2532875 · Cited by 563
Software testing and analysis: process, principles, and techniques
(2008) · Choice Reviews Online · doi:10.5860/choice.46-0935 · Cited by 340
Self-citationBasiri, A., Behnam, N., de Rooij, R., Hochstein, L., Kosewski, L., Reynolds, J., & Rosenthal, C. (2016) · IEEE Software · doi:10.1109/ms.2016.60 · Cited by 285 · cited 1×
“For example, in NetFlix the concept of Chaos engineering for online testing the robustness of their cloud services has been established a decade ago and still continues to be expanded [9].”
BackgroundOnline Controlled Experiments and A/B Testing
Kohavi, R., & Longbotham, R. (2017) · Encyclopedia of Machine Learning and Data Mining · doi:10.1007/978-1-4899-7687-1_891 · Cited by 179 · cited 1×
“• Exclusion Criterion 2: studies purely focusing on techniques other than testing, most notably we exclude works centered on: runtime verification [68], or debugging, even approaches for debugging in the field such as [65], or continuous experimentation via user surveys [90] and A/B testing [57], or usability evaluati…”
BackgroundBugRedux: reproducing field failures for in-house debugging
Jin, W., & Orso, A. (2012) · doi:10.5555/2337223.2337279 · Cited by 131
A Metamorphic Testing Approach for Online Testing of Service-Oriented Software Applications
Chan, W. K., Cheung, S.-C., & Leung, K. R. P. H. (2007) · International Journal of Web Services Research · doi:10.4018/jwsr.2007040103 · Cited by 127 · cited 7×
“Domain-dependent ([63]; [53]; [24]; [75]; [69]; [22]; [62]; [27]; [95]; [78]; [37]; [2]; [3]; [71]; [70]; [12]; [46]; [72]; [80]; [6]; [38]; [98]; [31]; [18]; [20]; [10]; [51]; [93]; [32]; [8]; [76]; [86]; [41]; [21]; [48])”
MethodInfluential\nTest Generation Based on Symbolic Specifications
Frantzen, L., Tretmans, J., & Willemse, T. A. C. (2005) · Radboud Repository (Radboud University) · doi:10.1007/978-3-540-31848-4_1 · Cited by 125 · cited 2×
“Their auditing approach selects test cases at runtime guided by both the behavioral specification of the SUT (i.e., Symbolic Transition Systems (STS) [36]) and the output returned by the SUT at each interaction.”
Background(2003) · International Conference on Parallel Architectures and Compilation Techniques (PACT) · doi:10.1109/pact.2003.1237995 · Cited by 125
Proceedings of the 1995 ACM symposium on Applied computing - SAC '95
(1995) · doi:10.1145/315891 · Cited by 123
The disappearing boundary between development-time and run-time
Baresi, L., & Ghezzi, C. (2010) · doi:10.1145/1882362.1882367 · Cited by 112 · cited 2×
“The impossibility of dealing with all the faults using classic in-house testing approaches raised interest in testing software systems in the field, by crossing the border between in-house validation and field execution [7].”
BackgroundAutomatic system testing of programs without test oracles
Murphy, C., Shen, K., & Kaiser, G. (2009) · Proceedings of the eighteenth international symposium on Software testing and analysis · doi:10.1145/1572272.1572295 · Cited by 84 · cited 9×
“Domain-dependent ([63]; [53]; [24]; [75]; [69]; [22]; [62]; [27]; [95]; [78]; [37]; [2]; [3]; [71]; [70]; [12]; [46]; [72]; [80]; [6]; [38]; [98]; [31]; [18]; [20]; [10]; [51]; [93]; [32]; [8]; [76]; [86]; [41]; [21]; [48])”
BackgroundMethodInfluentialOntology-Based Test Case Generation for Testing Web Services
Wang, Y., Bai, X., Li, J., & Huang, R. (2007) · International Symposium on Autonomous Decentralized Systems · doi:10.1109/isads.2007.54 · Cited by 81 · cited 6×
“, method calls/service invocations) [24]; [17]; [75]; [30]; [82]; [48]; [3]; [80]; [47]; [31]; [83]; [99]; [5]; [102]; [51]; [100]; [32]; [55]; [45]”
BackgroundMethodInfluentialTowards run-time adaptation of test cases for self-adaptive systems in the face of uncertainty
Fredericks, E. M., DeVries, B., & Cheng, B. H. C. (2014) · Proceedings of the 9th International Symposium on Software Engineering for Adaptive and Self-Managing Systems · doi:10.1145/2593929.2593937 · Cited by 72 · cited 9×
“Domain-dependent ([63]; [53]; [24]; [75]; [69]; [22]; [62]; [27]; [95]; [78]; [37]; [2]; [3]; [71]; [70]; [12]; [46]; [72]; [80]; [6]; [38]; [98]; [31]; [18]; [20]; [10]; [51]; [93]; [32]; [8]; [76]; [86]; [41]; [21]; [48])”
BackgroundInfluentialOntology-Based Test Modeling and Partition Testing of Web Services
Bai, X., Lee, S., Tsai, W., & Chen, Y. (2008) · IEEE International Conference on Web Services (ICWS) · doi:10.1109/icws.2008.111 · Cited by 70 · cited 5×
“, method calls/service invocations) [24]; [17]; [75]; [30]; [82]; [48]; [3]; [80]; [47]; [31]; [83]; [99]; [5]; [102]; [51]; [100]; [32]; [55]; [45]”
BackgroundMethodInfluentialOnline testing with model programs
Veanes, M., Campbell, C., Schulte, W., & Tillmann, N. (2005) · Proceedings of the 10th European software engineering conference held jointly with 13th ACM SIGSOFT international symposium on Foundations of software engineering · doi:10.1145/1081706.1081751 · Cited by 66 · cited 1×
“, on-thefly model based testing [96], or those works that dynamically adjust the testing strategy using testing results, such as [19].”
MethodQuality Assurance of Software Applications Using the In Vivo Testing Approach
Murphy, C., Kaiser, G. E., Vo, I. D., & Chu, M. (2009) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2009.18 · Cited by 65 · cited 10×
“Domain-dependent ([63]; [53]; [24]; [75]; [69]; [22]; [62]; [27]; [95]; [78]; [37]; [2]; [3]; [71]; [70]; [12]; [46]; [72]; [80]; [6]; [38]; [98]; [31]; [18]; [20]; [10]; [51]; [93]; [32]; [8]; [76]; [86]; [41]; [21]; [48])”
BackgroundMethodInfluentialVARAN the Unbelievable: An Efficient N-version Execution Framework
Hošek, P., & Cadar, C. (2015) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/2694344.2694390 · Cited by 64 · cited 10×
“, method calls/service invocations) [24]; [17]; [75]; [30]; [82]; [48]; [3]; [80]; [47]; [31]; [83]; [99]; [5]; [102]; [51]; [100]; [32]; [55]; [45]”
MethodInfluentialA Systematic Review on Cloud Testing
Bertolino, A., Angelis, G. D., Gallego, M., García, B., Gortázar, F., Lonetti, F., & Marchetti, E. (2019) · ACM Computing Surveys · doi:10.1145/3331447 · Cited by 62 · cited 1×
“• Exclusion Criterion 3: studies that move testing away from the developer’s laboratory to improve scalability and elasticity, but with no specific focus on field operation: for example, we exclude works proposing Testing-as-a-Service and Cloud testing solutions [11].”
BackgroundSelf-citationCollaborative Testing of Web Services
Zhu, H., & Zhang, Y. (2010) · IEEE Transactions on Services Computing · doi:10.1109/tsc.2010.54 · Cited by 56 · cited 7×
“a client or another system/module ([94]; [103]; [8]); [104]) a tester ( [21]; [70]; [103]); [104]) the run-time infrastructure/container ([70]; [16]; [18])”
MethodInfluentialUsage-Based Online Testing for Proactive Adaptation of Service-Based Applications
Sammodi, O., Metzger, A., Franch, X., Oriol, M., Marco, J., & Pohl, K. (2011) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac.2011.81 · Cited by 52 · cited 11×
“Some domain-independent oracles rely on QoS attributes of the SUT, such as general performance metrics, like response time [89, 70, 23], or execution duration and availability [83].”
BackgroundMethodInfluentialRößler, J., Zeller, A., Fraser, G., Zamfir, C., & Candea, G. (2013) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2013.18 · Cited by 51 · cited 2×
“Related to ex-vivo testing, there are a number of works that exploit failure/crash data from field executions for the generation of test cases that would help in reproducing the failures for in-house debugging (e.g., References [29, 50, 52, 88, 92]).”
BackgroundWeb Services Regression Testing
Penta, M. D., Bruno, M., Esposito, G., Mazza, V., & Canfora, G. (2007) · Springer eBooks · doi:10.1007/978-3-540-72912-9_8 · Cited by 50
Crowdsourced software testing: A systematic literature review
Alyahya, S. (2020) · Information and Software Technology · doi:10.1016/j.infsof.2020.106363 · Cited by 48
Skoll: A Process and Infrastructure for Distributed Continuous Quality Assurance
Porter, A., Yılmaz, C., Memon, A. M., Schmidt, D. C., & Natarajan, B. (2007) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2007.70719 · Cited by 46
The MORABIT Approach to Runtime Component Testing
Suliman, D., Paech, B., Borner, L., Atkinson, C., Brenner, D., Merdes, M., & Malaka, R. (2006) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac.2006.169 · Cited by 45 · cited 9×
“new module discovered or deployed ([53]; [12]; [40]; [94]; [64]; [14]; [83]; [102]; [105]) new module bound ([53]; [66]; [94]; [73]; [80]; [6]; [33]) module change or fault ([66]; [40]; [64]; [14]; [73]; [80]; [6]; [51]; [93]; [32]) module removed ([64])”
BackgroundMethodInfluentialOptimal and adaptive testing for software reliability assessment
Cai, K., Li, Y., & Liu, K. (2004) · Information and Software Technology · doi:10.1016/j.infsof.2004.07.006 · Cited by 44
Continuous Experimentation: Challenges, Implementation Techniques, and Current Research
Schermann, G., Cito, J., & Leitner, P. (2018) · IEEE Software · doi:10.1109/ms.2018.111094748 · Cited by 44 · cited 1×
“• Exclusion Criterion 2: studies purely focusing on techniques other than testing, most notably we exclude works centered on: runtime verification [68], or debugging, even approaches for debugging in the field such as [65], or continuous experimentation via user surveys [90] and A/B testing [57], or usability evaluati…”
BackgroundSOA Test Governance: Enabling Service Integration Testing across Organization and Technology Borders
Bertolino, A., & Polini, A. (2009) · International Conference on Software Testing, Verification and Validation Workshops · doi:10.1109/icstw.2009.39 · Cited by 43 · cited 1×
“In the context of field testing, a governance framework provides a setting to execute and control testing activities, and establishes policies that guide the decision of when, how much, and how to test: Test Governance concerns the establishment and enforcement of policies, procedures, notations and tools that are req…”
BackgroundSelf-citationSafe and efficient runtime testing framework applied in dynamic and distributed systems
Lahami, M., Krichen, M., & Jmaïel, M. (2016) · Science of Computer Programming · doi:10.1016/j.scico.2016.02.002 · Cited by 43 · cited 8×
“new module discovered or deployed ([53]; [12]; [40]; [94]; [64]; [14]; [83]; [102]; [105]) new module bound ([53]; [66]; [94]; [73]; [80]; [6]; [33]) module change or fault ([66]; [40]; [64]; [14]; [73]; [80]; [6]; [51]; [93]; [32]) module removed ([64])”
MethodInfluentialTowards pro-active adaptation with confidence
Metzger, A., Sammodi, O., Pohl, K., & Rzepka, M. (2010) · International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS) · doi:10.1145/1808984.1808987 · Cited by 41 · cited 8×
“new module discovered or deployed ([53]; [12]; [40]; [94]; [64]; [14]; [83]; [102]; [105]) new module bound ([53]; [66]; [94]; [73]; [80]; [6]; [33]) module change or fault ([66]; [40]; [64]; [14]; [73]; [80]; [6]; [51]; [93]; [32]) module removed ([64])”
BackgroundMethodInfluentialA Framework for Service-Oriented Testing of Web Services
Zhu, H. (2006) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac.2006.95 · Cited by 41 · cited 7×
“a client or another system/module ([94]; [103]; [8]); [104]) a tester ( [21]; [70]; [103]); [104]) the run-time infrastructure/container ([70]; [16]; [18])”
MethodInfluentialTowards Self-Testing in Autonomic Computing Systems
King, T. M., Babich, D., Alava, J., Clarke, P., & Stevens, R. (2007) · International Symposium on Autonomous Decentralized Systems · doi:10.1109/isads.2007.75 · Cited by 41 · cited 10×
“Yet other approaches rely on periodic inspection of attributes of the system and the environment, such as system resource consumption [54] and network data [51], [100].”
BackgroundMethodInfluentialBringing white-box testing to Service Oriented Architectures through a Service Oriented Approach
Bartolini, C., Bertolino, A., Elbaum, S. G., & Marchetti, E. (2010) · Journal of Systems and Software · doi:10.1016/j.jss.2010.10.024 · Cited by 38 · cited 8×
“Domain-dependent ([63]; [53]; [24]; [75]; [69]; [22]; [62]; [27]; [95]; [78]; [37]; [2]; [3]; [71]; [70]; [12]; [46]; [72]; [80]; [6]; [38]; [98]; [31]; [18]; [20]; [10]; [51]; [93]; [32]; [8]; [76]; [86]; [41]; [21]; [48])”
BackgroundMethodInfluentialSelf-citationAn Exploratory Study of Field Failures
Gazzola, L., Mariani, L., Pastore, F., & Pezzè, M. (2017) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre.2017.10 · Cited by 38 · cited 1×
“[39] identifies several categories of field failures, and provides empirical evidence of the unavoidability of failures in the field even for mature and well-tested software systems.”
BackgroundSelf-citationElbaum, S. G., & Hardojo, M. (2004) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/1007512.1007522 · Cited by 37 · cited 6×
“User defined ([25]) Unspecified ([43]; [81]; [82]; [42]; [66]; [77]; [40]; [79]; [64]; [14]; [60]; [1]; [54]; [16]; [58]; [73] [59]; [47]; [99]; [5]; [13]; [105]; [101]; [33]; [100] [34]; [103]; [104]; [61]; [55]; [45]; [94]; [28])”
BackgroundInfluentialAn Integrated Self-Testing Framework for Autonomic Computing Systems
King, T. M., Ramirez, A. E., Cruz, R., & Clarke, P. (2007) · Journal of Computers · doi:10.4304/jcp.2.9.37-49 · Cited by 37 · cited 10×
“, method calls/service invocations) [24]; [17]; [75]; [30]; [82]; [48]; [3]; [80]; [47]; [31]; [83]; [99]; [5]; [102]; [51]; [100]; [32]; [55]; [45]”
BackgroundMethodInfluentialThe PLASTIC Framework and Tools for Testing Service-Oriented Applications
Bertolino, A., Angelis, G. D., Frantzen, L., & Polini, A. (2009) · Lecture notes in computer science · doi:10.1007/978-3-540-95888-8_5 · Cited by 36 · cited 11×
“Domain-dependent ([63]; [53]; [24]; [75]; [69]; [22]; [62]; [27]; [95]; [78]; [37]; [2]; [3]; [71]; [70]; [12]; [46]; [72]; [80]; [6]; [38]; [98]; [31]; [18]; [20]; [10]; [51]; [93]; [32]; [8]; [76]; [86]; [41]; [21]; [48])”
BackgroundMethodInfluentialSelf-citationModel-Based Conformance Testing of WS-BPEL Compositions
Maâlej, A. J., Krichen, M., & Jmaïel, M. (2012) · IEEE Annual Computer Software and Applications Conference Workshops (COMPSACW) · doi:10.1109/compsacw.2012.86 · Cited by 35 · cited 3×
“online functional Choreographies and Service-based specifications [22, 2, 12, 6, 99, 5] Finite-State Models [71, 95, 18, 33, 20] Metamorphic Relations [21] Graph grammars [83] Event Interface [101] Fault Injection [102] Test planning and management [60, 1, 59, 14, 27, 43, 42] Adaptation and Reconfiguration [44, 53, 64…”
BackgroundMethodOnline Testing Framework for Web Services
Cao, T.-D., Félix, P., Castanet, R., & Berrada, I. (2010) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2010.11 · Cited by 35 · cited 6×
“Domain-dependent ([63]; [53]; [24]; [75]; [69]; [22]; [62]; [27]; [95]; [78]; [37]; [2]; [3]; [71]; [70]; [12]; [46]; [72]; [80]; [6]; [38]; [98]; [31]; [18]; [20]; [10]; [51]; [93]; [32]; [8]; [76]; [86]; [41]; [21]; [48])”
BackgroundMethodInfluentialTowards a TTCN-3 Test System for Runtime Testing of Adaptable and Distributed Systems
Lahami, M., Fakhfakh, F., Krichen, M., & Jmaïel, M. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-34691-0_7 · Cited by 34 · cited 14×
“Governance rules and policies are specified ([27];[14];[58];[59])”
BackgroundMethodInfluentialConfiguration Fuzzing for Software Vulnerability Detection
Dai, H., Murphy, C., & Kaiser, G. E. (2010) · International Conference on Availability, Reliability and Security (ARES) · doi:10.1109/ares.2010.22 · Cited by 34 · cited 9×
“Domain-dependent ([63]; [53]; [24]; [75]; [69]; [22]; [62]; [27]; [95]; [78]; [37]; [2]; [3]; [71]; [70]; [12]; [46]; [72]; [80]; [6]; [38]; [98]; [31]; [18]; [20]; [10]; [51]; [93]; [32]; [8]; [76]; [86]; [41]; [21]; [48])”
BackgroundInfluentialRuntime testing approach of structural adaptations for dynamic and distributed systems
Lahami, M., Krichen, M., & Jmaïel, M. (2015) · International Journal of Computer Applications in Technology · doi:10.1504/ijcat.2015.070489 · Cited by 32 · cited 15×
“Domain-dependent ([63]; [53]; [24]; [75]; [69]; [22]; [62]; [27]; [95]; [78]; [37]; [2]; [3]; [71]; [70]; [12]; [46]; [72]; [80]; [6]; [38]; [98]; [31]; [18]; [20]; [10]; [51]; [93]; [32]; [8]; [76]; [86]; [41]; [21]; [48])”
MethodInfluentialDynamic Reconfigurable Testing of Service-Oriented Architecture
Bai, X., Xu, D., & Dai, G. (2007) · Proceedings - International Computer Software & Applications Conference · doi:10.1109/compsac.2007.106 · Cited by 32 · cited 8×
“Domain-dependent ([63]; [53]; [24]; [75]; [69]; [22]; [62]; [27]; [95]; [78]; [37]; [2]; [3]; [71]; [70]; [12]; [46]; [72]; [80]; [6]; [38]; [98]; [31]; [18]; [20]; [10]; [51]; [93]; [32]; [8]; [76]; [86]; [41]; [21]; [48])”
BackgroundMethodInfluentialWhitening SOA Testing via Event Exposure
Ye, C., & Jacobsen, H. (2013) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2013.20 · Cited by 32 · cited 8×
“User defined ([25]) Unspecified ([43]; [81]; [82]; [42]; [66]; [77]; [40]; [79]; [64]; [14]; [60]; [1]; [54]; [16]; [58]; [73] [59]; [47]; [99]; [5]; [13]; [105]; [101]; [33]; [100] [34]; [103]; [104]; [61]; [55]; [45]; [94]; [28])”
BackgroundMethodInfluentialSoftware testing with an operational profile: OP definition
Smidts, C. S., Mutha, C., Rodríguez, M., & Gerber, M. J. (2014) · ACM Computing Surveys · doi:10.1145/2518106 · Cited by 31
A benchmark-based evaluation of search-based crash reproduction
Soltani, M., Derakhshanfar, P., Devroey, X., & Arie Van Deursen, A. (2019) · Empirical Software Engineering · doi:10.1007/s10664-019-09762-1 · Cited by 31
Lahami, M., Krichen, M., Bouchakwa, M., & Jmaïel, M. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-34691-0_9 · Cited by 30 · cited 3×
“online functional Choreographies and Service-based specifications [22, 2, 12, 6, 99, 5] Finite-State Models [71, 95, 18, 33, 20] Metamorphic Relations [21] Graph grammars [83] Event Interface [101] Fault Injection [102] Test planning and management [60, 1, 59, 14, 27, 43, 42] Adaptation and Reconfiguration [44, 53, 64…”
MethodA TTCN-3 based online test and validation platform for Internet services
Deussen, P. H., Din, G., & Schieferdecker, I. (2003) · doi:10.1109/isads.2003.1193946 · Cited by 27 · cited 7×
“Domain-dependent ([63]; [53]; [24]; [75]; [69]; [22]; [62]; [27]; [95]; [78]; [37]; [2]; [3]; [71]; [70]; [12]; [46]; [72]; [80]; [6]; [38]; [98]; [31]; [18]; [20]; [10]; [51]; [93]; [32]; [8]; [76]; [86]; [41]; [21]; [48])”
MethodInfluentialEnhancing Service Federation Trustworthiness through Online Testing
Bertolino, A., Angelis, G. D., Kellomaki, S., & Polini, A. (2011) · Computer · doi:10.1109/mc.2011.227 · Cited by 27 · cited 9×
“Compensation ([13]) COW (copy-on-write) file system ([24])”
BackgroundMethodInfluentialSelf-citationPreemptive Regression Testingof Workflow-Based Web Services
Mei, L., Chan, W. K., Tse, T. H., Jiang, B., & Zhai, K. (2014) · IEEE Transactions on Services Computing · doi:10.1109/tsc.2014.2322621 · Cited by 27
Reproducing Field Failures for Programs with Complex Grammar-Based Input
Kifetew, F. M., Jin, W., Tiella, R., Orso, A., & Tonella, P. (2014) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2014.29 · Cited by 25 · cited 2×
“Related to ex-vivo testing, there are a number of works that exploit failure/crash data from field executions for the generation of test cases that would help in reproducing the failures for in-house debugging (e.g., References [29, 50, 52, 88, 92]).”
BackgroundSelf-citationAutomatic runtime recovery via error handler synthesis
Gu, T., Sun, C. P., Ma, X., Lü, J., & Su, Z. (2016) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/2970276.2970360 · Cited by 24 · cited 12×
“custom events ([31]) unchecked Java exception ([44])”
BackgroundMethodInfluentialArchitecture support for runtime integration and verification of component-based Systems of Systems
Gonzalez, A., Piel, É., & Groß, H.-G. (2008) · IEEE/ACM International Conference on Automated Software Engineering Workshops (ASEW) · doi:10.1109/asew.2008.4686292 · Cited by 23 · cited 10×
“Domain-dependent ([63]; [53]; [24]; [75]; [69]; [22]; [62]; [27]; [95]; [78]; [37]; [2]; [3]; [71]; [70]; [12]; [46]; [72]; [80]; [6]; [38]; [98]; [31]; [18]; [20]; [10]; [51]; [93]; [32]; [8]; [76]; [86]; [41]; [21]; [48])”
MethodInfluentialAn Approach to Facilitate Reliability Testing of Web Services Components
Zhang, J. (2005) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre.2004.4 · Cited by 22 · cited 11×
“Supports both ([30]; [26]; [83]; [102]; [23]; [25])”
BackgroundMethodInfluentialMonitoring the service-based system lifecycle with SALMon
Oriol, M., Franch, X., & Marco, J. (2015) · Expert Systems with Applications · doi:10.1016/j.eswa.2015.03.027 · Cited by 22 · cited 11×
“, method calls/service invocations) [24]; [17]; [75]; [30]; [82]; [48]; [3]; [80]; [47]; [31]; [83]; [99]; [5]; [102]; [51]; [100]; [32]; [55]; [45]”
BackgroundInfluentialAutonomous decentralized system test technique
Kawano, K., Orimo, M., & Mori, K. (2003) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/cmpsac.1989.65053 · Cited by 19 · cited 12×
“Domain-dependent ([63]; [53]; [24]; [75]; [69]; [22]; [62]; [27]; [95]; [78]; [37]; [2]; [3]; [71]; [70]; [12]; [46]; [72]; [80]; [6]; [38]; [98]; [31]; [18]; [20]; [10]; [51]; [93]; [32]; [8]; [76]; [86]; [41]; [21]; [48])”
BackgroundMethodInfluentialSafe Runtime Validation of Behavioral Adaptations in Autonomic Software
King, T. M., Allen, A. A., Cruz, R., & Clarke, P. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-23496-5_3 · Cited by 18 · cited 10×
“Domain-dependent ([63]; [53]; [24]; [75]; [69]; [22]; [62]; [27]; [95]; [78]; [37]; [2]; [3]; [71]; [70]; [12]; [46]; [72]; [80]; [6]; [38]; [98]; [31]; [18]; [20]; [10]; [51]; [93]; [32]; [8]; [76]; [86]; [41]; [21]; [48])”
BackgroundMethodInfluentialReducing Verification Effort in Component-Based Software Engineering through Built-In Testing
Brenner, D., Atkinson, C., Paech, B., Malaka, R., Merdes, M., & Suliman, D. (2006) · IEEE International Enterprise Distributed Object Computing Conference (EDOC) · doi:10.1109/edoc.2006.44 · Cited by 18
Automated generation of adaptive test plans for self-adaptive systems
Fredericks, E. M., & Cheng, B. H. C. (2015) · Software Engineering for Adaptive and Self-Managing Systems · doi:10.5555/2821357.2821385 · Cited by 18
Search‐based crash reproduction using behavioural model seeding
Derakhshanfar, P., Devroey, X., Perrouin, G., Zaidman, A., & Arie Van Deursen, A. (2020) · Software Testing Verification and Reliability · doi:10.1002/stvr.1733 · Cited by 18 · cited 2×
“Related to ex-vivo testing, there are a number of works that exploit failure/crash data from field executions for the generation of test cases that would help in reproducing the failures for in-house debugging (e.g., References [29, 50, 52, 88, 92]).”
BackgroundAn Extensible Framework for Online Testing of Choreographed Services
Ali, M., Angelis, F. D., Fani', D., Bertolino, A., Angelis, G. D., & Polini, A. (2014) · Computer · doi:10.1109/mc.2013.407 · Cited by 17
Self-citationTest Coverage of Data-Centric Dynamic Compositions in Service-Based Systems
Hummer, W., Raz, O., Shehory, O. M., Leitner, P., & Dustdar, S. (2011) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2011.55 · Cited by 17 · cited 10×
“, method calls/service invocations) [24]; [17]; [75]; [30]; [82]; [48]; [3]; [80]; [47]; [31]; [83]; [99]; [5]; [102]; [51]; [100]; [32]; [55]; [45]”
BackgroundInfluentialAutomatic Discovery of Web Services Based on Dynamic Black-Box Testing
Park, Y., Jung, W.-S., Lee, B., & Wu, C. (2009) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac.2009.24 · Cited by 16 · cited 15×
“Supports both ([30]; [26]; [83]; [102]; [23]; [25])”
BackgroundMethodInfluentialLessons learned at 208K: Towards debugging millions of cores
Lee, G. L., Ahn, D. H., Arnold, D. C., de Supinski, B. R., LeGendre, M., Miller, B. P., Schulz, M., & Liblit, B. (2008) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1109/sc.2008.5218557 · Cited by 16 · cited 1×
“• Exclusion Criterion 2: studies purely focusing on techniques other than testing, most notably we exclude works centered on: runtime verification [68], or debugging, even approaches for debugging in the field such as [65], or continuous experimentation via user surveys [90] and A/B testing [57], or usability evaluati…”
BackgroundEnabling Proactive Adaptation through Just-in-Time Testing of Conversational Services
Dranidis, D., Metzger, A., & Kourtesis, D. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-17694-4_6 · Cited by 15 · cited 6×
“new module discovered or deployed ([53]; [12]; [40]; [94]; [64]; [14]; [83]; [102]; [105]) new module bound ([53]; [66]; [94]; [73]; [80]; [6]; [33]) module change or fault ([66]; [40]; [64]; [14]; [73]; [80]; [6]; [51]; [93]; [32]) module removed ([64])”
MethodInfluentialTowards Change Propagating Test Models in Autonomic and Adaptive Systems
Akour, M., Jaidev, A., & King, T. M. (2011) · IEEE International Conference and Workshop on the Engineering of Computer-Based Systems (ECBS) · doi:10.1109/ecbs.2011.23 · Cited by 15 · cited 4×
“online functional Choreographies and Service-based specifications [22, 2, 12, 6, 99, 5] Finite-State Models [71, 95, 18, 33, 20] Metamorphic Relations [21] Graph grammars [83] Event Interface [101] Fault Injection [102] Test planning and management [60, 1, 59, 14, 27, 43, 42] Adaptation and Reconfiguration [44, 53, 64…”
BackgroundMethodInfluentialTesting of data‐centric and event‐based dynamic service compositions
Hummer, W., Raz, O., Shehory, O. M., Leitner, P., & Dustdar, S. (2013) · Software Testing Verification and Reliability · doi:10.1002/stvr.1493 · Cited by 14
Achieving Dependable Component Bindings in Dynamic Adaptive Systems - A Runtime Testing Approach
Niebuhr, D., Rausch, A., Klein, C., Reichmann, J., & Schmid, R. (2009) · IEEE International Conference on Self-Adaptive and Self-Organizing Systems (SASO) · doi:10.1109/saso.2009.40 · Cited by 14
On coping with real-time software dynamic inconsistency by built-in tests
Wang, Y., King, G. A., Patel, D. A., Patel, S. D., & Dorling, A. (1999) · Annals of Software Engineering · doi:10.1023/a:1018990322378 · Cited by 13 · cited 3×
“Field testing was first considered an opportunity to deal with failures that were hard or too costly to reproduce in the laboratory, with a few studies in the nineties addressing autonomic systems [51, 100], and real-time issues [98].”
BackgroundMethodCONFU: Configuration Fuzzing Testing Framework for Software Vulnerability Detection
Dai, H., Murphy, C., & Kaiser, G. E. (2010) · International Journal of Secure Software Engineering · doi:10.4018/jsse.2010070103 · Cited by 13 · cited 10×
“Supports both ([30]; [26]; [83]; [102]; [23]; [25])”
BackgroundInfluentialda Costa, A. D., Nunes, C., da Silva, V. T., Fonseca, B., & de Lucena, C. J. P. (2010) · ACM Symposium on Applied Computing · doi:10.1145/1774088.1774280 · Cited by 12 · cited 12×
“Supports both ([30]; [26]; [83]; [102]; [23]; [25])”
MethodInfluentialA self-testing autonomic container
Stevens, R., Parsons, B., & King, T. M. (2007) · ACM Southeast Regional Conference (ACMSE) · doi:10.1145/1233341.1233343 · Cited by 12 · cited 7×
“Domain-dependent ([63]; [53]; [24]; [75]; [69]; [22]; [62]; [27]; [95]; [78]; [37]; [2]; [3]; [71]; [70]; [12]; [46]; [72]; [80]; [6]; [38]; [98]; [31]; [18]; [20]; [10]; [51]; [93]; [32]; [8]; [76]; [86]; [41]; [21]; [48])”
MethodInfluentialTest Isolation Policy for Safe Runtime Validation of Evolvable Software Systems
Lahami, M., & Krichen, M. (2013) · doi:10.1109/wetice.2013.62 · Cited by 12 · cited 10×
“Governance rules and policies are specified ([27];[14];[58];[59])”
BackgroundMethodInfluentialA Dependable Online Testing and Upgrade Architecture for Real-Time Embedded Systems
Lee, K., & Sha, L. (2006) · IEEE International Conference on Embedded and Real-Time Computing Systems and Applications (RTCSA) · doi:10.1109/rtcsa.2005.8 · Cited by 12 · cited 8×
“new module discovered or deployed ([53]; [12]; [40]; [94]; [64]; [14]; [83]; [102]; [105]) new module bound ([53]; [66]; [94]; [73]; [80]; [6]; [33]) module change or fault ([66]; [40]; [64]; [14]; [73]; [80]; [6]; [51]; [93]; [32]) module removed ([64])”
BackgroundMethodInfluential(role)CAST: A FRAMEWORK FOR ON-LINE SERVICE TESTING
Angelis, G. D., Bertolino, A., & Polini, A. (2011) · International Conference on Web Information Systems and Technologies · doi:10.5220/0003340500130018 · Cited by 12 · cited 21×
“Supports both ([30]; [26]; [83]; [102]; [23]; [25])”
BackgroundInfluentialSelf-citationDesign of BDI Agent for Adaptive Performance Testing of Web Services
Ma, B., Chen, B., Bai, X., & Huang, J. (2010) · Asia-Pacific Conference on Quality Software (APAQS) · doi:10.1109/qsic.2010.69 · Cited by 11
Murphy, C., Vaughan, M., Ilahi, W., & Kaiser, G. E. (2010) · doi:10.1145/1808266.1808269 · Cited by 11 · cited 8×
“offline functional IO Data Pattern [81] Metamorphic Relations [10, 76] Built-in Tests [94, 98, 75, 54, 55, 93] Test planning and management [43, 42, 77] Adaptation and Reconfiguration [62, 64]”
BackgroundMethodInfluentialTowards Ex Vivo Testing of MapReduce Applications
Morán, J., Bertolino, A., de la Riva, C., & Tuya, J. (2017) · IEEE International Conference on Software Quality, Reliability and Security (QRS) · doi:10.1109/qrs.2017.17 · Cited by 11 · cited 5×
“[74], where the authors employ data collected from actual user requests for the testing of MapReduce distributed data processing applications, thus overcoming the costs of building huge amount of data as those used in these applications.”
BackgroundMethodInfluentialSelf-citationData-flow integration testing adapted to runtime evolution in component-based systems
Piel, É., & Sánchez, A. G. (2009) · doi:10.1145/1596495.1596499 · Cited by 10
StealthTest: Low Overhead Online Software Testing Using Transactional Memory
Bobba, J., Xiong, W., Yen, L., Hill, M. D., & Wood, D. A. (2009) · International Conference on Parallel Architectures and Compilation Techniques (PACT) · doi:10.1109/pact.2009.15 · Cited by 10 · cited 18×
“, method calls/service invocations) [24]; [17]; [75]; [30]; [82]; [48]; [3]; [80]; [47]; [31]; [83]; [99]; [5]; [102]; [51]; [100]; [32]; [55]; [45]”
BackgroundMethodInfluentialAn Environment to Support Structural Testing of Autonomous Vehicles
Neves, V. D. O., Delamaro, M. E., & Masiero, P. C. (2014) · Brazilian Symposium on Computing System Engineering (SBESC) · doi:10.1109/sbesc.2014.27 · Cited by 9 · cited 8×
“: produced by the OS/Network/others) ([3]; [28]; [79])”
BackgroundInfluentialRiTMO: A Method for Runtime Testability Measurement and Optimisation
Sánchez, A. G., Piel, É., & Groß, H.-G. (2009) · Proceedings of the ... International Conference on Quality Software/Proceedings · doi:10.1109/qsic.2009.56 · Cited by 9 · cited 6×
“offline functional IO Data Pattern [81] Metamorphic Relations [10, 76] Built-in Tests [94, 98, 75, 54, 55, 93] Test planning and management [43, 42, 77] Adaptation and Reconfiguration [62, 64]”
MethodInfluentialOnline Robustness Testing of Distributed Embedded Systems: An Industrial Approach
Alnawasreh, K., Pelliccione, P., Hao, Z., Range, M., & Bertolino, A. (2017) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip.2017.17 · Cited by 9 · cited 12×
“Domain-dependent ([63]; [53]; [24]; [75]; [69]; [22]; [62]; [27]; [95]; [78]; [37]; [2]; [3]; [71]; [70]; [12]; [46]; [72]; [80]; [6]; [38]; [98]; [31]; [18]; [20]; [10]; [51]; [93]; [32]; [8]; [76]; [86]; [41]; [21]; [48])”
BackgroundInfluentialSelf-citationProceedings of the 12th Iberoamerican Conference on Software Engineering (CibSE’09)
Brogi, A., Araujo, J., & Anaya, R. (2009) · Cited by 9
A Test Automation Framework for Collaborative Testing of Web Service Dynamic Compositions
Zhu, H., & Zhang, Y. (2013) · Springer eBooks · doi:10.1007/978-1-4614-7535-4_8 · Cited by 8 · cited 4×
“Evolution [58, 61, 63, 66] Shadow Instances for V&V [40, 45] SOA and Component-Based Testing [8, 16, 81, 103, 104, 105] Built-In and Pre-Existing Test Cases [30, 31, 41, 51, 81, 100] SOA Online Testing [80] Evolution [23, 37, 58, 61, 63] Distributed Quality Assurance [87] a priori we do not expect that new, ad-hoc tec…”
BackgroundMethodInfluentialOn-line software test technique based on autonomous decentralized system
Yamamoto, H., Yoshizawa, A., Kawano, K., Mori, K., Oshima, K., & Kawamura, T. (2002) · IEEE Workshop on Future Trends of Distributed Computing Systems (FTDCS) · doi:10.1109/ftdcs.1993.344143 · Cited by 8 · cited 11×
“Yet other approaches rely on periodic inspection of attributes of the system and the environment, such as system resource consumption [54] and network data [51], [100].”
BackgroundMethodInfluentialOn-line testing for application software of widely distributed system
Nishijima, E., Yamamoto, H., Kawano, K., Fujiwara, K., & Oshima, K. (2002) · IEEE Symposium on Reliable Distributed Systems (SRDS) · doi:10.1109/reldis.1996.559698 · Cited by 8 · cited 14×
“offline functional IO Data Pattern [81] Metamorphic Relations [10, 76] Built-in Tests [94, 98, 75, 54, 55, 93] Test planning and management [43, 42, 77] Adaptation and Reconfiguration [62, 64]”
MethodInfluentialDynamic Test Reconfigurationfor Composite Web Services
Cooray, M. B., Hamlyn-Harris, J., & Merkel, R. (2014) · IEEE Transactions on Services Computing · doi:10.1109/tsc.2014.2312953 · Cited by 7
VALIDATION AND VERIFICATION POLICIES FOR GOVERNANCE OF SERVICE CHOREOGRAPHIES
Angelis, G. D., Bertolino, A., & Polini, A. (2012) · International Conference on Web Information Systems and Technologies · doi:10.5220/0003936200580070 · Cited by 6 · cited 7×
“Domain-dependent ([63]; [53]; [24]; [75]; [69]; [22]; [62]; [27]; [95]; [78]; [37]; [2]; [3]; [71]; [70]; [12]; [46]; [72]; [80]; [6]; [38]; [98]; [31]; [18]; [20]; [10]; [51]; [93]; [32]; [8]; [76]; [86]; [41]; [21]; [48])”
BackgroundMethodInfluentialSelf-citationA Distributed Test Architecture for Adaptable and Distributed Real-Time Systems.
Lahami, M., Krichen, M., & Jmaïel, M. (2011) · Cited by 6
Governance Policies for Verification and Validation of Service Choreographies
Bertolino, A., Angelis, G. D., & Polini, A. (2013) · Lecture notes in business information processing · doi:10.1007/978-3-642-36608-6_6 · Cited by 5 · cited 9×
“new module discovered or deployed ([53]; [12]; [40]; [94]; [64]; [14]; [83]; [102]; [105]) new module bound ([53]; [66]; [94]; [73]; [80]; [6]; [33]) module change or fault ([66]; [40]; [64]; [14]; [73]; [80]; [6]; [51]; [93]; [32]) module removed ([64])”
MethodInfluentialSelf-citationSchedulable Online Testing Framework for Real-Time Embedded Applications in VM
Goh, O., & Lee, Y.-H. (2007) · Springer eBooks · doi:10.1007/978-3-540-77092-3_63 · Cited by 5 · cited 13×
“new module discovered or deployed ([53]; [12]; [40]; [94]; [64]; [14]; [83]; [102]; [105]) new module bound ([53]; [66]; [94]; [73]; [80]; [6]; [33]) module change or fault ([66]; [40]; [64]; [14]; [73]; [80]; [6]; [51]; [93]; [32]) module removed ([64])”
BackgroundMethodInfluentialMetamorphic Runtime Checking of Applications Without Test Oracles
Bell, J., Murphy, C., & Kaiser, G. E. (2014) · doi:10.7916/d8j9655p · Cited by 5
Runtime testing framework for improving quality in dynamic service-based systems
Lahami, M., Krichen, M., & Jmaïel, M. (2013) · doi:10.1145/2489300.2489335 · Cited by 5 · cited 11×
“Domain-dependent ([63]; [53]; [24]; [75]; [69]; [22]; [62]; [27]; [95]; [78]; [37]; [2]; [3]; [71]; [70]; [12]; [46]; [72]; [80]; [6]; [38]; [98]; [31]; [18]; [20]; [10]; [51]; [93]; [32]; [8]; [76]; [86]; [41]; [21]; [48])”
MethodInfluentialNeves, V. D. O., Delamaro, M. E., & Masiero, P. C. (2016) · International Journal of Embedded Systems · doi:10.1504/ijes.2016.080388 · Cited by 5
An on-line test platform for component-based systems
Deussen, P. H., Din, G., & Schieferdecker, I. (2002) · doi:10.1109/sew.2002.1199455 · Cited by 4 · cited 7×
“Supports both ([30]; [26]; [83]; [102]; [23]; [25])”
BackgroundMethodInfluentialA model for usability testing in challenging environments
Luostarinen, R., Järvinen, R., Määttä, J., & Manner, J. (2016) · International Conference on Military Communications and Information Systems (ICMCIS) · doi:10.1109/icmcis.2016.7496557 · Cited by 2 · cited 11×
“Domain-dependent ([63]; [53]; [24]; [75]; [69]; [22]; [62]; [27]; [95]; [78]; [37]; [2]; [3]; [71]; [70]; [12]; [46]; [72]; [80]; [6]; [38]; [98]; [31]; [18]; [20]; [10]; [51]; [93]; [32]; [8]; [76]; [86]; [41]; [21]; [48])”
BackgroundMethodInfluentialA runtime-testing method for integer overflow detection based on metamorphic relations
Hui, Z., Huang, S., & Ji, M.-Y. (2016) · Journal of Intelligent & Fuzzy Systems · doi:10.3233/jifs-169076 · Cited by 2 · cited 10×
“Domain-dependent ([63]; [53]; [24]; [75]; [69]; [22]; [62]; [27]; [95]; [78]; [37]; [2]; [3]; [71]; [70]; [12]; [46]; [72]; [80]; [6]; [38]; [98]; [31]; [18]; [20]; [10]; [51]; [93]; [32]; [8]; [76]; [86]; [41]; [21]; [48])”
BackgroundInfluentialNeves, V. D. O., Delamaro, M. E., & Masiero, P. C. (2016) · International Journal of Embedded Systems · doi:10.1504/ijes.2016.10001345 · Cited by 2 · cited 3×
“Field testing approaches that target embedded applications consider interactions with hardware and environment, and focus primarily on ex-vivo testing [78, 28, 79] that can be safely performed in-house.”
BackgroundDevelopment of Test Agents for Automated Dynamic Testing of UNIWAY
Lee, S.-H., & Na, J.-C. (2017) · Lecture notes in electrical engineering · doi:10.1007/978-981-10-5041-1_109 · Cited by 1 · cited 8×
“Manual ( [67]; [69])) Random ([17, 75])”
BackgroundInfluential
Research works printed but not linked: 5
- Smt. J. Sasi Bhanu , A. Vinaya Babu , and P. Trimurthy . 2015 . A comprehensive architecture for dynamic evolution of online testing of an embedded system . J. Theor. Appl. Inf. Technol. 80 , 1 (Oct. 2015), 160--172. http://www.jatit.org/volumes/Vol80No1/17Vol80No1.pdf Smt. J. Sasi Bhanu, A. Vinaya Babu, and P. Trimurthy. 2015. A comprehensive architecture for dynamic evolution of online testing of an embedded system. J. Theor. Appl. Inf. Technol. 80, 1 (Oct. 2015), 160--172. http://www.jatit.org/volumes/Vol80No1/17Vol80No1.pdf
- 10.5555/829505.830124 · doi:10.5555/829505.830124
- 10.5555/850965.854686 · doi:10.5555/850965.854686
- Alberto Gonzalez-Sanchez , Eric Piel , Hans-Gerhard Gross , and Arjan J. C . van Gemund. 2010. Runtime testability in dynamic high-availability component-based systems . In Proceedings of the 2nd International Conference on Advances in System Testing and Validation Lifecycle (VALID’10) . IEEE, 37--42. DOI:https://doi.org/10.1109/VALID. 2010 .13 10.1109/VALID.2010.13 Alberto Gonzalez-Sanchez, Eric Piel, Hans-Gerhard Gross, and Arjan J. C. van Gemund. 2010. Runtime testability in dynamic high-availability component-based systems. In Proceedings of the 2nd International Conference on Advances in System Testing and Validation Lifecycle (VALID’10). IEEE, 37--42. DOI:https://doi.org/10.1109/VALID.2010.13 · doi:10.1109/valid
- Jüri Vain , Leonidas Tsiopoulos , Vyacheslav Kharchenko , Apneet Kaur , Maksim Jenihhin , and Jaan Raik . 2017 . Multi-fragment Markov model guided online test generation for MPSoC . In Proceedings of the 3rd International Workshop on Theory of Reliability and Markov Modeling for Information Technologies (TheRMIT’17) . 594--607. Jüri Vain, Leonidas Tsiopoulos, Vyacheslav Kharchenko, Apneet Kaur, Maksim Jenihhin, and Jaan Raik. 2017. Multi-fragment Markov model guided online test generation for MPSoC. In Proceedings of the 3rd International Workshop on Theory of Reliability and Markov Modeling for Information Technologies (TheRMIT’17). 594--607.
Cited by
Latent class analysis in PLS-SEM: A review and recommendations for future applications
Sarstedt, M., Radomir, L., Moisescu, O. I., & Ringle, C. M. (2022) · Journal of Business Research · doi:10.1016/j.jbusres.2021.08.051 · Cited by 319
Usability research in educational technology: a state-of-the-art systematic review
Lu, J., Schmidt, M., Lee, M., & Huang, R. (2022) · Educational Technology Research and Development · doi:10.1007/s11423-022-10152-6 · Cited by 143
Generative AI for self-adaptive systems: State of the art and research roadmap
Li, J., Zhang, M., Li, N., Weyns, D., Jin, Z., & Tei, K. (2024) · ACM Transactions on Autonomous and Adaptive Systems · doi:10.1145/3686803 · Cited by 77
Software Engineering by and for Humans in an AI Era
Abrahão, S., Grundy, J., Pezzè, M., Storey, M.-A., & Tamburri, D. A. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3715111 · Cited by 37
“SE might play a more active role in the process of specifying and evaluating such systems in the future.”
Self-citationOnline testing of RESTful APIs: promises and challenges
Martin-Lopez, A., Segura, S., & Ruiz–Cortés, A. (2022) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3540250.3549144 · Cited by 37
“Online testing refers to the testing performed on production software [60, 63], as opposed to offline testing, which is done on a development environment, before the software is released.”
Method
Metadata
- Authors
- Bertolino, A., Braione, P., Angelis, G. D., Gazzola, L., Kifetew, F., Mariani, L., Orrù, M., Pezzè, M., Pietrantuono, R., Russo, S., & Tonella, P.
- Affiliations
- Istituto di Scienza e Tecnologie dell'Informazione "Alessandro Faedo" (IT)
- University of Milano-Bicocca (IT)
- Istituto di Analisi dei Sistemi ed Informatica Antonio Ruberti (IT)
- Fondazione Bruno Kessler (IT)
- Constructor Institute (CH)
- University of Naples Federico II (IT)
- Università della Svizzera italiana (CH)
- ORCID
- Venue
- ACM Computing Surveys
- Published
- 2021-05-25
- Volume
- 54
- Issue
- 5
- Pages
- 1-39
- ISSN
- 0360-0300 (ISSN-L)
- 1557-7341
- OpenAlex
- W3165293486
- Cited by
- 37
- References
- 106
- Citations by year
- 2026: 3 · 2025: 11 · 2024: 5 · 2023: 10 · 2022: 7 · 2021: 1
- FWCI
- 9.54
- Type
- journal article
- Language
- en
- Open access
- Locations
- ACM Computing Surveys (publishedVersion)
- BOA (University of Milano-Bicocca) (submittedVersion)
- ISTI Open Portal (submittedVersion, other-oa)
- Funders
- Italian MIUR PRIN 2015 (GAUSS - Prot. 2015KWREMX)
- European Research Council (PRECRIME - Advanced Grant 2017 Agreement n. 787703)
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Keywords
- field testing, software testing, software defects, fault detection, software quality
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.