Model-based testing for software safety: a systematic mapping study
Open accessThe DOI matches its Crossref record; 73 of the 73 references it cites were found in a registry or an index.
Abstract
Testing safety-critical systems is crucial since a failure or malfunction may result in death or serious injuries to people, equipment, or environment. An important challenge in testing is the derivation of test cases that can identify the potential faults. Model-based testing adopts models of a system under test and/or its environment to derive test artifacts. This paper aims to provide a systematic mapping study to identify, analyze, and describe the state-of-the-art advances in model-based testing for software safety. The systematic mapping study is conducted as a multi-phase study selection process using the published literature in major software engineering journals and conference proceedings. We reviewed 751 papers and 36 of them have been selected as primary studies to answer our research questions. Based on the analysis of the data extraction process, we discuss the primary trends and approaches and present the identified obstacles. This study shows that model-based testing can provide important benefits for software safety testing. Several solution directions have been identified, but further research is critical for reliable model-based testing approach for safety.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Therefore, we constructed a review protocol before conducting the review, based on the guidelines from Kitchenham and Charters [4], Ali et al. [31], and Gurbuz and Tekinerdogan [5].”
From Automation of systematic literature reviews: A systematic literature review (2021) · cites this paper as [5]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Grading quality of evidence and strength of recommendations
(2004) · BMJ · doi:10.1136/bmj.328.7454.1490 · Cited by 8,477 · cited 4×
“In this work, we used the definitions from the Grading of Recommendations Assessment, Development and Evaluation (GRADE) (Atkins et al. 2004) working group which is developed for grading the quality of evidence and strength of recommendations.”
MethodInfluentialExperimentation in Software Engineering
Wohlin, C., Runeson, P., Höst, M., Ohlsson, M. C., Regnell, B., & Wesslén, A. (2012) · doi:10.1007/978-3-642-29044-2 · Cited by 4,460 · cited 2×
“We consider the categorization in Wohlin et al. (2012) to extract research method during the review process.”
BackgroundMethodGuidelines for conducting systematic mapping studies in software engineering: An update
Petersen, K., Vakkalanka, S., & Kuzniarz, L. (2015) · Information and Software Technology · doi:10.1016/j.infsof.2015.03.007 · Cited by 2,669 · cited 2×
“We carry out the SMS to provide an overview of existing research directions regarding the MBT for software safety by following the guidelines and process proposed by (Petersen et al. 2015).”
MethodJensen, K. (1987) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-47919-2_10 · Cited by 1,744
Experimentation in software engineering
(2013) · Choice Reviews Online · doi:10.5860/choice.50-5018 · Cited by 1,517
The guide to the Software Engineering Body of Knowledge
Bourque, P., Dupuis, R., Abran, A., Moore, J. W., & Tripp, L. (1999) · IEEE Software · doi:10.1109/52.805471 · Cited by 1,083
A taxonomy of model‐based testing approaches
Utting, M., Pretschner, A., & Legeard, B. (2012) · Software Testing, Verification and Reliability · doi:10.1002/stvr.456 · Cited by 632
Identifying relevant studies in software engineering
Zhang, H., Babar, M. A., & Tell, P. (2011) · Information and Software Technology · doi:10.1016/j.infsof.2010.12.010 · Cited by 557
Introduction to automata theory, languages, and computation, 2nd edition
Hopcroft, J. E., Motwani, R., & Ullman, J. D. (2001) · ACM SIGACT News · doi:10.1145/568438.568455 · Cited by 510
Guide to the Software Engineering Body of Knowledge : 2004 Version
Abran, A., Moore, J. W., Bourque, P., Dupuis, R., & Tripp, L. L. (2005) · Medical Entomology and Zoology · Cited by 375
A survey on model-based testing approaches: a systematic review
Dias Neto, A. C., Subramanyan, R., Vieira, M., & Travassos, G. H. (2007) · Proceedings of the 1st ACM international workshop on Empirical assessment of software engineering languages and technologies: held in conjunction with the 22nd IEEE/ACM International Conference on Automated Software Engineering (ASE) 2007 · doi:10.1145/1353673.1353681 · Cited by 306 · cited 1×
“In Dias Neto et al. (2007), they perform a systematic review on model-based testing approaches.”
Rafi, D. M., Moses, K. R. K., Petersen, K., & Mäntylä, M. (2012) · International Workshop on Automation of Software Test (AST) · doi:10.1109/iwast.2012.6228988 · Cited by 170 · cited 2×
“…has the advantage that the primary studies can also be examined by other reviewers, and this will support the reliability and reproducibility of our SMS. 5 Related work Rafi et al. (2012) provide a systematic literature review and survey on benefits and limitations of automated software testing.”
BackgroundMethodTesting with model checkers: a survey
Fraser, G., Wotawa, F., & Ammann, P. E. (2009) · Software Testing, Verification and Reliability · doi:10.1002/stvr.402 · Cited by 159 · cited 1×
“Model checkers are formal verification tools which are capable of providing counter examples to violated properties (Fraser et al. 2009).”
BackgroundA taxonomy of model-based testing
Utting, M., Pretschner, A., & Legeard, B. (2006) · Research Commons (University of Waikato) · Cited by 155
Rafi, D. M., Moses, K. R. K., Petersen, K., & Mäntylä, M. (2012) · Automation of Software Test · doi:10.5555/2663608.2663616 · Cited by 139
JUMBL: a tool for model-based statistical testing
Prowell, S. (2003) · Proceedings of the ... Annual Hawaii International Conference on System Sciences/Proceedings of the Annual Hawaii International Conference on System Sciences · doi:10.1109/hicss.2003.1174916 · Cited by 68 · cited 1×
“Usage Model Builder Library) tool (Prowell 2003) which is a model-based testing tool for statistical testing in order to generate test cases and test scripts.”
MethodAutomated test generation using model checking: an industrial evaluation
Enoiu, E. P., Čaušević, A., Ostrand, T. J., Weyuker, E. J., Sundmark, D., & Pettersson, P. (2014) · International Journal on Software Tools for Technology Transfer · doi:10.1007/s10009-014-0355-9 · Cited by 66
Model-Based Coverage-Driven Test Suite Generation for Software Product Lines
Cichos, H., Oster, S., Lochau, M., & Schürr, A. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-24485-8_31 · Cited by 60
Risk-Based Testing of Safety-Critical Embedded Systems Driven by Fault Tree Analysis
Kloos, J., Hussain, T., & Eschbach, R. (2011) · IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) · doi:10.1109/icstw.2011.90 · Cited by 50
A Concept for Testing Robustness and Safety of the Context-Aware Behaviour of Autonomous Systems
Micskei, Z., Szatmári, Z., Olah, J., & Majzik, I. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-30947-2_55 · Cited by 49
Automatic test case generation from Simulink/Stateflow models using model checking
Mohalik, S. K., Gadkari, A. A., Yeolekar, A., Shashidhar, K. C., & Ramesh, S. (2013) · Software Testing Verification and Reliability · doi:10.1002/stvr.1489 · Cited by 47
Model-Based Test Suite Generation for Function Block Diagrams Using the UPPAAL Model Checker
Enoiu, E. P., Sundmark, D., & Pettersson, P. (2013) · IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) · doi:10.1109/icstw.2013.27 · Cited by 44
Formal Analysis and Testing of Real-Time Automotive Systems Using UPPAAL Tools
Kim, J. H., Larsen, K. G., Nielsen, B., Mikučionis, M., & Olsen, P. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-19458-5_4 · Cited by 38
Using Model Checking to Generate Fault Detecting Tests
Gargantini, A. (2007) · Lecture notes in computer science · doi:10.1007/978-3-540-73770-4_11 · Cited by 34
Rigorous development process of a safety-critical system: from ASM models to Java code
Arcaini, P., Gargantini, A., & Riccobene, E. (2015) · International Journal on Software Tools for Technology Transfer · doi:10.1007/s10009-015-0394-x · Cited by 33
Choosing a Test Modeling Language: A Survey
Hartman, A., Katara, M., & Olvovsky, S. (2007) · Springer eBooks · doi:10.1007/978-3-540-70889-6_16 · Cited by 32 · cited 1×
“In the study of Hartman et al. (2007), the authors conduct a survey on the different types of languages used to define test models.”
BackgroundA Global Algorithm for Model-Based Test Suite Generation
Hessel, A., & Pettersson, P. (2007) · Electronic Notes in Theoretical Computer Science · doi:10.1016/j.entcs.2007.08.005 · Cited by 32 · cited 1×
“[5] uses CoVeR tool (Hessel and Pettersson 2007a) The second most used solution approach is model checking.”
MethodModel-driven test generation for system level validation
Mathaikutty, D. A., Ahuja, S., Dingankar, A. T., & Shukla, S. K. (2007) · Proceedings · doi:10.1109/hldvt.2007.4392792 · Cited by 28
Environment behavior models for automation of testing and assessment of system safety
Auguston, M., Michael, J. B., & Shing, M. (2006) · Information and Software Technology · doi:10.1016/j.infsof.2006.03.005 · Cited by 27
Towards Model-Driven V&V assessment of railway control systems
Marrone, S., Flammini, F., Mazzocca, N., Nardone, R., & Vittorini, V. (2014) · International Journal on Software Tools for Technology Transfer · doi:10.1007/s10009-014-0320-7 · Cited by 25
Formal Model-Based Test for AUTOSAR Multicore RTOS
Fang, L., Kitamura, T., Do, T. B. N., & Ohsaki, H. (2012) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2012.105 · Cited by 24
Feature Interaction Aware Test Case Generation for Embedded Control Systems
Lochau, M., & Goltz, U. (2010) · Electronic Notes in Theoretical Computer Science · doi:10.1016/j.entcs.2010.12.013 · Cited by 23
Aichernig, B. K., Brandl, H., Jöbstl, E., & Krenn, W. (2011) · ACM SIGSOFT Software Engineering Notes · doi:10.1145/1921532.1921559 · Cited by 22 · cited 2×
“[17] use Argos and Ulysses (Aichernig et al. 2011) tools to generate OOAS models from UML diagrams.”
MethodFormal Modeling and Verification of Cyber-Physical Systems
Drechsler, R., & Kühne, U. (2015) · Springer Fachmedien Wiesbaden · doi:10.1007/978-3-658-09994-7 · Cited by 18
Cover - A Real-Time Test Case Generation Tool
Hessel, A., & Pettersson, P. (2007) · Cited by 17
Towards a Framework for Explicit Platform-Based Transformations
Thomas, F., Delatour, J., Terrier, F., & G, S. (2008) · IEEE International Symposium on Real-Time Distributed Computing (ISORC) · doi:10.1109/isorc.2008.64 · Cited by 17 · cited 1×
“However, in Thomas et al. (2008), the authors propose a UML profile on architectural level aim to provide a tool for formal verification and validation techniques such as model checking and runtime verification.”
BackgroundDebugging with Timed Automata Mutations
Aichernig, B. K., Hörmaier, K., & Lorber, F. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-10506-2_4 · Cited by 17
Mapping UML to Labeled Transition Systems for Test-Case Generation
Krenn, W., Schlick, R., & Aichernig, B. K. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-17071-3_10 · Cited by 16
Toward DO-178B-compliant Test Models
Stallbaum, H., & Rzepka, M. (2010) · doi:10.1109/modevva.2010.21 · Cited by 16
Krenn, W., Schlick, R., & Aichernig, B. K. (2009) · Formal Methods · Cited by 15
Systematic scenario test case generation for nuclear safety systems
Tseng, W.-H., & Fan, C.-F. (2012) · Information and Software Technology · doi:10.1016/j.infsof.2012.08.016 · Cited by 15
Stateflow to Extended Finite Automata Translation
Li, M., & Kumar, R. (2011) · IEEE Annual Computer Software and Applications Conference Workshops (COMPSACW) · doi:10.1109/compsacw.2011.11 · Cited by 15 · cited 1×
“In [9], models are described by using StateFlow which has been adopted from StateChart and allows hierarchical modeling of discrete behaviors consisting of parallel and exclusive decompositions, which makes it challenging to capture and translate into formal models (Li and Kumar 2011).”
MethodZheng, W., Liang, C., Wang, R., & Kong, W. (2014) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2014.2320552 · Cited by 14
Constraint-based test generation for automotive operating systems
Choi, Y., & Byun, T. (2015) · Software & Systems Modeling · doi:10.1007/s10270-014-0449-6 · Cited by 13
Development of a Framework for Automated Systematic Testing of Safety-Critical Embedded Systems
Kandl, S., Kirner, R., & Puschner, P. P. (2006) · Workshop on Intelligent Solutions in Embedded Systems · doi:10.1109/wises.2006.329116 · Cited by 12
Leskovec, J., Rajaraman, A., & Ullman, J. D. (2014) · Cambridge University Press eBooks · doi:10.1017/cbo9781139924801.011 · Cited by 11 · cited 1×
“First, the term frequency, inverse document frequency (TF*IDF) algorithm (Leskovec et al. 2014), was operated on the titles and abstracts of the QGS papers.”
MethodDeriving Usage Model Variants for Model-Based Testing: An Industrial Case Study
Samih, H., Guen, H. L., Bogusch, R., Acher, M., & Baudry, B. (2014) · IEEE International Conference on Engineering of Complex Computer Systems (ICECCS) · doi:10.1109/iceccs.2014.19 · Cited by 10
Yu, G., Xu, Z. W., & Du, J. W. (2009) · International Forum on Information Technology and Applications · doi:10.1109/ifita.2009.18 · Cited by 10
Kloos, J., & Eschbach, R. (2010) · Electronic Notes in Theoretical Computer Science · doi:10.1016/j.entcs.2010.05.009 · Cited by 9
Fail-safe testing of safety-critical systems: a case study and efficiency analysis
Gario, A., Andrews, A. K. A., & Hagerman, S. (2015) · Software Quality Journal · doi:10.1007/s11219-015-9283-5 · Cited by 9
Model-based test cases generation for Onboard system
Lv, J., Li, K., Wei, G., Tang, T., Li, C., & Zhao, W. (2013) · International Symposium on Autonomous Decentralized Systems · doi:10.1109/isads.2013.6513433 · Cited by 8
Generating test case chains for reactive systems
Schrammel, P., Melham, T., & Kroening, D. (2014) · International Journal on Software Tools for Technology Transfer · doi:10.1007/s10009-014-0358-6 · Cited by 8
Model Based Testing and Abstract Interpretation in the Railway Signaling Context
Grasso, D., Fantechi, A., Ferrari, A., Becheri, C., & Bacherini, S. (2010) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2010.44 · Cited by 8
Test Specification Patterns for Automatic Generation of Test Sequences
Gentile, U., Marrone, S., Mele, G., Nardone, R., & Peron, A. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-10702-8_12 · Cited by 6
Guide to the software engineering body of knowledge stakeholder issues and intended usages
Dupuis, R., & Bourque, P. (1999) · doi:10.1109/sess.1999.767664 · Cited by 5
Model-Based Safety Test Automation of Safety-Critical Software
Yu, G., & Xu, Z. W. (2010) · doi:10.1109/cise.2010.5676883 · Cited by 5
A Systematic Testing Approach for Autonomous Mobile Robots Using Domain-Specific Languages
Proetzsch, M., Zimmermann, F., Eschbach, R., Kloos, J., & Berns, K. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-16111-7_36 · Cited by 5
Generating Scenarios by Multi-Object Checking
Kollmann, M., & Hon, Y. M. (2007) · Electronic Notes in Theoretical Computer Science · doi:10.1016/j.entcs.2007.08.006 · Cited by 4
Scalable Incremental Test-case Generation from Large Behavior Models
Aichernig, B. K., Ničković, D., & Tiran, S. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-21215-9_1 · Cited by 3
A Systematic Approach to Requirements Driven Test Generation for Safety Critical Systems
Wilkinson, T., Butler, M. J., & Colley, J. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-12214-4_4 · Cited by 2
Towards generation of efficient test cases from UML/OCL models for complex safety-critical systems
Herzner, W., Schlick, R., Schütz, W., Brandl, H., & Krenn, W. (2010) · e+i Elektrotechnik und Informationstechnik · doi:10.1007/s00502-010-0741-2 · Cited by 2
Development of a Framework for Automated Systematic Testing of Safety-Critical Embedded Systems
Kandl, S. (2006) · doi:10.1109/wises.2006.237154 · Cited by 2
Barbero (2014) · doi:10.1016/j · Cited by 1
Computer Safety, Reliability, and Security
Bondavalli, A., & Di Giandomenico, F. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-10506-2
Blanchette, J. C., & Kosmatov, N. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-21215-9
(2007) · Lecture notes in computer science · doi:10.1007/978-3-540-73770-4
Hardware and Software, Verification and Testing
(2007) · Lecture notes in computer science · doi:10.1007/978-3-540-70889-6
Formal Methods for Components and Objects
A, F. S. D. B., Leuschel, M., Hallerstede, S. H., & Bonsangue, M. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-17071-3
Agent and Multi-Agent Systems. Technologies and Applications
Ježić, G., Jain, L. C., Howlett, R. J., Nguyên, N. T., & Kušek, M. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-30947-2
KI 2010: Advances in Artificial Intelligence
Dillmann, R., Schultz, T., Hanebeck, U. D., & Beyerer, J. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-16111-7
Model-Based Safety and Assessment
Ortmeier, F. 1., & München, I. 2. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-12214-4
Jensen, K. (2006) · Lecture Notes in Computer Science · doi:10.1007/bfb0046842
Cited by
Predictive maintenance using digital twins: A systematic literature review
van Dinter, R., Tekinerdogan, B., & Catal, C. (2022) · Information and Software Technology · doi:10.1016/j.infsof.2022.107008 · Cited by 540
“While developing the quality assessment criteria, the summary quality checklist for quantitative and qualitative studies has been adopted, as proposed by [7] and [36].”
MethodAutomation of systematic literature reviews: A systematic literature review
van Dinter, R., Tekinerdogan, B., & Catal, C. (2021) · Information and Software Technology · doi:10.1016/j.infsof.2021.106589 · Cited by 432
“The success of SLR in evidence-based medicine triggered various other research areas to adopt similar SLR approaches [5].”
BackgroundMethodInfluentialObstacles and features of Farm Management Information Systems: A systematic literature review
Tummers, J., Kassahun, A., & Tekinerdoğan, B. (2019) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2018.12.044 · Cited by 153
Digital twin for ship life-cycle: A critical systematic review
Mauro, F., & Kana, A. A. (2022) · Ocean Engineering · doi:10.1016/j.oceaneng.2022.113479 · Cited by 157
“To this end, the quality assessment proposed by Kietchenham and Charters (2007) and Gurbuz and Tekinerdogan (2018) is a suitable example, requiring answering the following quality index (QI): QI-1 Are the aims of the study clearly stated?”
MethodWomen’s Participation in Open Source Software: A Survey of the Literature
Trinkenreich, B., Wiese, I., Sarma, A., Gerosa, M., & Steinmacher, I. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3510460 · Cited by 100
“The GRADE approach is widely used by secondary studies in software engineering to evaluate the strength of evidence found in the primary studies [1, 2, 30, 47, 48, 96].”
Method
Metadata
- Authors
- Gurbuz, H. G., & Tekinerdogan, B.
- ORCID
- Gurbuz, Havva Gulay
- Venue
- Software Quality Journal
- Published
- 2017-09-25
- Volume
- 26
- Issue
- 4
- Pages
- 1327-1372
- ISSN
- 0963-9314 (ISSN-L)
- 1573-1367
- OpenAlex
- W2759114566
- Cited by
- 78
- References
- 64
- Citations by year
- 2026: 5 · 2025: 11 · 2024: 8 · 2023: 12 · 2022: 14 · 2021: 13 · 2020: 5 · 2019: 6 · 2018: 4
- FWCI
- 13.60
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- Locations
- Software Quality Journal (publishedVersion, cc-by)
- Socio-Environmental Systems Modeling (acceptedVersion, cc-by)
- Data Archiving and Networked Services (DANS) (submittedVersion, cc-by)
- Funders
- Wageningen University
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Engineering › Safety, Risk, Reliability and Quality › Safety Systems Engineering in Autonomy
- Physical Sciences › Computer Science › Software › Model-Driven Software Engineering Techniques
- Keywords
- model-based testing, software safety, safety-critical systems, test case derivation, fault detection, software testing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.