A Statistical Analysis of Operational Profile Driven Testing
Abstract
The amount of faults in software products is constantly increasing and it is not feasible to remove all faults. Within many organizations, operational profiles are used to focus the test activities. Prior art has claimed that when reliability growth tests are allocated using operational profiles, the reliability level will be the maximum practically achievable, but this claim has never been proven nor rejected. In this paper a statistical model is introduced that captures the relation between faults and failures. Using the model the effects of using operational profiles for reliability growth testing is analyzed. The analysis shows that the benefits of using operational profiles depend on a number of factors such as the number of faults and the failure probability. The results show that using operational profiles for reliability growth testing is beneficial, however not when a very high reliability is obtained, due to the effect of over testing.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Cites
PRODUCT DESIGN AND DEVELOPMENT
Ulrich, K. T., & Eppinger, S. D. (2018) · doi:10.4337/9781784718152.00017 · Cited by 5,528 · see citing works
Software Product Line Engineering
Pohl, K., Böckle, G., & van der Linden, F. (2005) · Springer eBooks · doi:10.1007/3-540-28901-1 · Cited by 3,409 · see citing works
Handbook of software reliability engineering
Lyu, M. R.-T. (1996) · McGraw-Hill, Inc. eBooks · Cited by 1,481 · see citing works
Product design and development
Gibson, W. M. (2022) · IWA Publishing eBooks · doi:10.2166/9781789061840_0019 · Cited by 849 · see citing works
Why Do Computers Stop and What Can Be Done About It?
Gray, J. (2024) · doi:10.6084/m9.figshare.27043420 · Cited by 665 · see citing works
Software Product Lines in Action: The Best Industrial Practice in Product Line Engineering
van der Linden, F. J., Schmid, K., & Rommes, E. (2007) · Springer eBooks · Cited by 552 · see citing works
Software reliability engineering
(1972) · Microelectronics Reliability · doi:10.1016/0026-2714(72)90036-4 · Cited by 497 · see citing works
Software Product Lines in Action
van der Linden, F., Schmid, K., & Rommes, E. (2007) · Springer eBooks · doi:10.1007/978-3-540-71437-8 · Cited by 485 · see citing works
Knight, J. (2012) · Chapman & Hall/CRC eBooks · doi:10.1201/b11667-14 · Cited by 320 · see citing works
Statistical testing of software based on a usage model
Walton, G. H., Poore, J. H., & Trammell, C. J. (1995) · Software Practice and Experience · doi:10.1002/spe.4380250106 · Cited by 164 · see citing works
Software Reliability Engineering
Tian, J. (2005) · doi:10.1002/0471722324.ch22 · Cited by 72 · see citing works
Powell, D. (1991) · Springer eBooks · doi:10.1007/978-3-642-84696-0_14 · Cited by 71 · see citing works
Reflections on industry trends and experimental research in dependability
Siewiorek, D. P., Chillarege, R., & Kalbarczyk, Z. (2004) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2004.20 · Cited by 53 · see citing works
How to measure the impact of specific development practices on fielded defect density
Neufelder, A. M. (2002) · doi:10.1109/issre.2000.885868 · Cited by 10 · see citing works
A decision support system for software reliability engineering strategy selection
Rus, I., & Collofello, J. S. (1999) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/cmpsac.1999.814311 · Cited by 9 · see citing works
Towards a weighted operational profile
Aggarwal, E. K., Kumar, M., Santurkar, V., et al. (1993) · ACM SIGSOFT Software Engineering Notes · doi:10.1145/157397.157402 · Cited by 1 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Fairness testing: testing software for discrimination
Galhotra, S., Brun, Y., & Meliou, A. (2017) · Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering · doi:10.1145/3106237.3106277 · Cited by 405 · see citing works
- Authors
- Hartmann, H.
- Affiliations
- University of Groningen (NL)
- Corresponding author
- Hartmann, Herman
- Venue
- 2016 IEEE International Conference on Software Quality, Reliability and Security Companion (QRS-C)
- Publisher
- IEEE
- Published
- 2016-08
- Pages
- 109-116
- OpenAlex
- W2521663729
- Cited by
- 4
- References
- 16
- Citations by year
- 2024: 1 · 2020: 1 · 2018: 1 · 2017: 1
- FWCI
- 0.49
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE International Conference on Software Quality, Reliability and Security Companion (QRS-C) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Software › Software Reliability and Analysis Research
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Engineering › Safety, Risk, Reliability and Quality › Reliability and Maintenance Optimization
- Keywords
- reliability growth testing, statistical models, unnecessary testing, failure probability, software reliability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
