The Oracle Problem in Software Testing: A Survey
Open accessThe DOI matches its Crossref record; 218 of the 236 references it cites were found in a registry or an index.
Abstract
Testing involves examining the behaviour of a system in order to discover potential faults. Given an input for a system, the challenge of distinguishing the corresponding desired, correct behaviour from potentially incorrect behavior is called the “test oracle problem”. Test oracle automation is important to remove a current bottleneck that inhibits greater overall test automation. Without test oracle automation, the human has to determine whether observed behaviour is correct. The literature on test oracles has introduced techniques for oracle automation, including modelling, specifications, contract-driven development and metamorphic testing. When none of these is completely adequate, the final source of test oracle information remains the human, who may be aware of informal specifications, expectations, norms and domain specific information that provide informal oracle guidance. All forms of test oracles, even the humble human, involve challenges of reducing cost and increasing benefit. This paper provides a comprehensive survey of current approaches to the test oracle problem and an analysis of trends in this important area of software testing research and practice.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures











Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The metric depends on rigorous testing, which is currently limited by the lack of test oracles [11].”
From A 2030 Roadmap for Software Engineering (2025) · cites this paper as [11]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Statecharts: a visual formalism for complex systems
Harel, D. (1987) · Science of Computer Programming · doi:10.1016/0167-6423(87)90035-9 · Cited by 6,751
An Axiomatic Basis for Computer Programming
Hoare, T. (1968) · Cited by 4,335
An axiomatic basis for computer programming
Hoare, C. A. R. (1969) · Communications of the ACM · doi:10.1145/363235.363259 · Cited by 4,002
Holzmann, G. J. (1997) · IEEE Transactions on Software Engineering · doi:10.1109/32.588521 · Cited by 3,803
Metaheuristics in combinatorial optimization: Overview and conceptual comparison
Blum, C., & Roli, A. (2003) · ACM Computing Surveys · doi:10.1145/937503.937505 · Cited by 3,236
Symbolic execution and program testing
King, J. C. (1976) · Communications of the ACM · doi:10.1145/360248.360252 · Cited by 3,031
Baader, F., & Nipkow, T. (1998) · Cambridge University Press eBooks · doi:10.1017/cbo9781139172752 · Cited by 2,466
Metaheuristics in Combinatorial Optimization
Gendreau, M., & Potvin, J.-Y. (2005) · Annals of Operations Research · doi:10.1007/s10479-005-3971-7 · Cited by 2,338
Learning regular sets from queries and counterexamples
Angluin, D. (1987) · Information and Computation · doi:10.1016/0890-5401(87)90052-6 · Cited by 2,114
DART: directed automated random testing
Godefroid, P., Klarlund, N., & Sen, K. (2005) · ACM SIGPLAN Notices · doi:10.1145/1064978.1065036 · Cited by 2,067
An Introduction to Computational Learning Theory
Kearns, M. J., & Vazirani, U. (1994) · doi:10.7551/mitpress/3897.001.0001 · Cited by 1,697
CUTE: a concolic unit testing engine for C
Sen, K., Marinov, D., & Agha, G. (2005) · Proceedings of the 10th European software engineering conference held jointly with 13th ACM SIGSOFT international symposium on Foundations of software engineering · doi:10.1145/1081706.1081750 · Cited by 1,651
DART: directed automated random testing
Godefroid, P., Klarlund, N., & Sen, K. (2005) · Proceedings of the 2005 ACM SIGPLAN conference on Programming language design and implementation · doi:10.1145/1065010.1065036 · Cited by 1,570
Testing Object-Oriented Systems: Models, Patterns, and Tools
Binder, R. V. (1999) · Cited by 1,549
Software abstractions: logic, language and analysis
(2007) · Choice Reviews Online · doi:10.5860/choice.44-2746 · Cited by 1,523
Assigning meanings to programs
Floyd, R. W. (1967) · Proceedings of Symposia in Applied Mathematics · doi:10.1090/psapm/019/0235771 · Cited by 1,408
Search‐based software test data generation: a survey
McMinn, P. (2004) · Software Testing, Verification and Reliability · doi:10.1002/stvr.294 · Cited by 1,368
Self-citationRegression testing minimization, selection and prioritization: a survey
Yoo, S., & Harman, M. (2012) · Software Testing, Verification and Reliability · doi:10.1002/stv.430 · Cited by 1,322
Self-citationAn empirical study of the reliability of UNIX utilities
Miller, B. P., Fredriksen, L., & So, B. (1990) · Communications of the ACM · doi:10.1145/96267.96279 · Cited by 1,281
Principles and methods of testing finite state machines-a survey
Lee, D., & Yannakakis, M. (1996) · Proceedings of the IEEE · doi:10.1109/5.533956 · Cited by 1,228
Practical Model-Based Testing: A Tools Approach
Utting, M., & Legeard, B. (2006) · Cited by 1,192
The Daikon system for dynamic detection of likely invariants
Ernst, M. D., Perkins, J. H., Guo, P. J., McCamant, S., Pacheco, C., Tschantz, M. S., & Xiao, C. (2007) · Science of Computer Programming · doi:10.1016/j.scico.2007.01.015 · Cited by 1,080
Simplifying and isolating failure-inducing input
Zeller, A., & Hildebrandt, R. (2002) · IEEE Transactions on Software Engineering · doi:10.1109/32.988498 · Cited by 1,079
Assigning Meanings to Programs
Floyd, R. W. (1993) · Studies in Cognitive Systems · doi:10.1007/978-94-011-1793-7_4 · Cited by 1,079
Dynamically discovering likely program invariants to support program evolution
Ernst, M. D., Cockrell, J., Griswold, W. G., & Notkin, D. (2001) · IEEE Transactions on Software Engineering · doi:10.1109/32.908957 · Cited by 1,050
The N-Version Approach to Fault-Tolerant Software
Avizienis, A. (1985) · IEEE Transactions on Software Engineering · doi:10.1109/tse.1985.231893 · Cited by 1,049
An Introduction to Input/Output Automata
Lynch, N. A., & Tuttle, M. R. (2026) · Formal Aspects of Computing · doi:10.1145/3788687 · Cited by 1,011
Formal verification of parallel programs
Keller, R. M. (1976) · Communications of the ACM · doi:10.1145/360248.360251 · Cited by 797
Search-based software engineering: Trends, techniques and applications
Harman, M., Mansouri, S. A., & Zhang, Y. (2012) · ACM Computing Surveys · doi:10.1145/2379776.2379787 · Cited by 777
Self-citationA System to Generate Test Data and Symbolically Execute Programs
Clarke, L. A. (1976) · IEEE Transactions on Software Engineering · doi:10.1109/tse.1976.233817 · Cited by 764
A specifier's introduction to formal methods
Wing, J. M. (1990) · Computer · doi:10.1109/2.58215 · Cited by 736
A methodology for controlling the size of a test suite
Harrold, M. J., Gupta, R., & Soffa, M. L. (1993) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/152388.152391 · Cited by 657
A taxonomy of model‐based testing approaches
Utting, M., Pretschner, A., & Legeard, B. (2012) · Software Testing, Verification and Reliability · doi:10.1002/stvr.456 · Cited by 632
On testing non-testable programs
Weyuker, E. J. (1982) · The Computer Journal · doi:10.1093/comjnl/25.4.465 · Cited by 606
Fraser, G., & Arcuri, A. (2013) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2012.14 · Cited by 586 · cited 1×
“Fraser and Arcuri [65] generate test suites for Java using their EvoSuite tool.”
MethodTest Generation with Inputs, Outputs and Repetitive Quiescence
Tretmans, J. (1996) · University of Twente Research Information · Cited by 579
Ali, S., Briand, L. C., Hemmati, H., & Panesar-Walawege, R. K. (2010) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2009.52 · Cited by 452
A systematic review of search-based testing for non-functional system properties
Afzal, W., Torkar, R., & Feldt, R. (2009) · Information and Software Technology · doi:10.1016/j.infsof.2008.12.005 · Cited by 447
Efficient detection of all pointer and array access errors
Austin, T. M., Breach, S. E., & Sohi, G. S. (1994) · Proceedings of the ACM SIGPLAN 1994 conference on Programming language design and implementation · doi:10.1145/178243.178446 · Cited by 427
Automatic generation of software behavioral models
Lorenzoli, D., Mariani, L., & Pezzè, M. (2008) · Proceedings of the 13th international conference on Software engineering - ICSE '08 · doi:10.1145/1368088.1368157 · Cited by 416
Functional documents for computer systems
Parnas, D. L., & Madey, J. (1995) · Science of Computer Programming · doi:10.1016/0167-6423(95)96871-j · Cited by 415
Houdini, an Annotation Assistant for ESC/Java
Flanagan, C., & Leino, K. R. M. (2001) · Lecture Notes in Computer Science · doi:10.1007/3-540-45251-6_29 · Cited by 405
PIE: a dynamic failure-based technique
Voas, J. M. (1992) · IEEE Transactions on Software Engineering · doi:10.1109/32.153381 · Cited by 402
Using formal specifications to support testing
Hierons, R. M., Bogdanov, K., Bowen, J. P., Cleaveland, R., Derrick, J., Dick, J., Gheorghe, M., Harman, M., Kapoor, K., Krause, P., et al. (2009) · ACM Computing Surveys · doi:10.1145/1459352.1459354 · Cited by 356
Self-citationEXE: Automatically Generating Inputs of Death
Cadar, C., Ganesh, V., Pawlowski, P. M., Dill, D. L., & Engler, D. R. (2008) · ACM Transactions on Information and System Security · doi:10.1145/1455518.1455522 · Cited by 342
A practical approach to programming with assertions
Rosenblum, D. S. (1995) · IEEE Transactions on Software Engineering · doi:10.1109/32.341844 · Cited by 334
SDL: Formal Object-Oriented Language for Communicating Systems
Sarma, A. C., Ellsberger, J., & Hogrefe, D. (1997) · Medical Entomology and Zoology · Cited by 323
Software testing based on formal specifications: a theory and a tool
Bernot, G., Claude Gaudel, M., & Marre, B. (1991) · Software Engineering Journal · doi:10.1049/sej.1991.0040 · Cited by 323
Search-Based Software Testing: Past, Present and Future
McMinn, P. (2011) · 2011 IEEE Fourth International Conference on Software Testing, Verification and Validation Workshops · doi:10.1109/icstw.2011.100 · Cited by 322
Self-citationThe ASTOOT approach to testing object-oriented programs
Doong, R.-K., & Frankl, P. G. (1994) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/192218.192221 · Cited by 321
Harman, M., Mansouri, S., & Zhang, Y. (2009) · UCL Discovery (University College London) · Cited by 313
Self-citationCUTE: A Concolic Unit Testing Engine for C
Sen, K., Marinov, D., & Agha, G. (2005) · doi:10.21236/ada482657 · Cited by 300
Regression testing minimization, selection and prioritization: a survey
Yoo, S., & Harman, M. (2012) · Software Testing, Verification and Reliability · doi:10.1002/stvr.430 · Cited by 297
Self-citationFuzzing for Software Security Testing and Quality Assurance
Takanen, A., DeMott, J. D., & Miller, C. (2008) · Cited by 288
Finite state description of communication protocols
Bochmann, G. V. (1978) · Computer Networks (1976) · doi:10.1016/0376-5075(78)90015-6 · Cited by 283
Peled, D., Vardi, M. Y., & Yannakakis, M. (1999) · IFIP advances in information and communication technology · doi:10.1007/978-0-387-35578-8_13 · Cited by 277
Experience with performance testing of software systems: issues, an approach, and case study
Weyuker, E. J., & Vokolos, F. I. (2000) · IEEE Transactions on Software Engineering · doi:10.1109/32.888628 · Cited by 255
Empirical studies of test‐suite reduction
Rothermel, G., Harrold, M. J., von Ronne, J., & Hong, C. (2002) · Software Testing, Verification and Reliability · doi:10.1002/stvr.256 · Cited by 253
Eclat: Automatic Generation and Classification of Test Inputs
Pacheco, C., & Ernst, M. D. (2005) · Lecture Notes in Computer Science · doi:10.1007/11531142_22 · Cited by 250
Formal specification: a roadmap
Lamsweerde, A. V. (2000) · Proceedings of the Conference on The Future of Software Engineering · doi:10.1145/336512.336546 · Cited by 249
Search Based Software Engineering: Techniques, Taxonomy, Tutorial
Harman, M., McMinn, P., de Souza, J. T., & Yoo, S. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-25231-0_1 · Cited by 245
Self-citationFinite-State Models for Logical Machines
(1969) · Students Quarterly Journal · doi:10.1049/sqj.1969.0041 · Cited by 239
Protocol testing: review of methods and relevance for software testing
Bochmann, G. V., & Petrenko, A. (1994) · Proceedings of the 1994 international symposium on Software testing and analysis - ISSTA '94 · doi:10.1145/186258.187153 · Cited by 229
A Simple and Practical Approach to Unit Testing: The JML and JUnit Way
Cheon, Y., & Leavens, G. T. (2002) · Lecture Notes in Computer Science · doi:10.1007/3-540-47993-7_10 · Cited by 228
An Overview of Anna, a Specification Language for Ada
Luckham, D. C., & Von Henke, F. W. (1985) · IEEE Software · doi:10.1109/ms.1985.230345 · Cited by 221
Data Abstraction, Implementation, Specification, and Testing
Gannon, J., McMullin, P., & Hamlet, R. (1981) · ACM Transactions on Programming Languages and Systems · doi:10.1145/357139.357140 · Cited by 221
An empirical investigation of the relationship between spectra differences and regression faults
Harrold, M. J., Rothermel, G., Sayre, K., Wu, R., & Yi, L. (2000) · Software Testing, Verification and Reliability · doi:10.1002/1099-1689(200009)10:3<171::aid-stvr209>3.0.co;2-j · Cited by 205
Turing, A. M. (1989) · The MIT Press eBooks · Cited by 188
CASL: the Common Algebraic Specification Language
Astesiano, E., Bidoit, M., Kirchner, H., Krieg-Brückner, B., Mosses, P. D., Sannella, D., & Tarlecki, A. (2002) · Theoretical Computer Science · doi:10.1016/s0304-3975(01)00368-1 · Cited by 183
Designing and comparing automated test oracles for GUI-based software applications
Xie, Q., & Memon, A. M. (2007) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/1189748.1189752 · Cited by 182
In black and white: an integrated approach to class-level testing of object-oriented programs
Chen, H. Y., Tse, T. H., Chan, F. T., & Chen, T. Y. (1998) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/287000.287004 · Cited by 175
Modelling systems: Practical tools and techniques in software development
(1999) · Computers & Mathematics with Applications · doi:10.1016/s0898-1221(99)90427-3 · Cited by 173
Using test oracles generated from program documentation
Peters, D. K., & Parnas, D. L. (1998) · IEEE Transactions on Software Engineering · doi:10.1109/32.667877 · Cited by 171
Performance, effectiveness, and reliability issues in software testing
Mathur, A. P. (2002) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/cmpsac.1991.170248 · Cited by 168
Correct Systems: Building a Business Process Solution
Holcombe, M., & Ipate, F. (1998) · Cited by 163
Eiffel: A language and environment for software engineering
Meyer, B. (1988) · Journal of Systems and Software · doi:10.1016/0164-1212(88)90022-2 · Cited by 153
Automated test oracles for GUIs
Memon, A. M., Pollack, M. E., & Soffa, M. L. (2000) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/355045.355050 · Cited by 149
Automatically repairing event sequence-based GUI test suites for regression testing
Memon, A. M. (2008) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/1416563.1416564 · Cited by 148
A practical flow-sensitive and context-sensitive C and C++ memory leak detector
Heine, D. L., & Lam, M. S. (2003) · ACM SIGPLAN Notices · doi:10.1145/780822.781150 · Cited by 147
Theoretical and Empirical Studies of Program Testing
Howden, W. E. (1978) · IEEE Transactions on Software Engineering · doi:10.1109/tse.1978.231514 · Cited by 144
A practical flow-sensitive and context-sensitive C and C++ memory leak detector
Heine, D. L., & Lam, M. S. (2003) · ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI) · doi:10.1145/781131.781150 · Cited by 143
TACCLE: a methodology for object-oriented software testing at the class and cluster levels
Chen, H. Y., Tse, T. H., & Chen, T. Y. (2001) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/366378.366380 · Cited by 141
Augmenting Automatically Generated Unit-Test Suites with Regression Oracle Checking
Xie, T. (2006) · Lecture notes in computer science · doi:10.1007/11785477_23 · Cited by 137
Discovering Algebraic Specifications from Java Classes
Henkel, J., & Ζ, A. D. (2003) · Lecture notes in computer science · doi:10.1007/978-3-540-45070-2_19 · Cited by 135
Regression Testing Minimisation, Selection and Prioritisation - A Survey
Yoo, S., & Harman, M. (2009) · UCL Discovery (University College London) · Cited by 134
Self-citationContext- and path-sensitive memory leak detection
Xie, Y., & Aiken, A. (2005) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/1081706.1081728 · Cited by 134
Test Case Prioritization Using Relevant Slices
Jeffrey, D., & Gupta, N. (2006) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac.2006.80 · Cited by 132
X-machines as a basis for dynamic system specification
Holcombe, M. (1988) · Software Engineering Journal · doi:10.1049/sej.1988.0009 · Cited by 131
Automatic synthesis of behavior protocols for composable web-services
Bertolino, A., Inverardi, P., Pelliccione, P., & Tivoli, M. (2009) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/1595696.1595719 · Cited by 130
Peled, D., Vardi, M. Y., & Yannakakis, M. (2001) · Journal of automata, languages and combinatorics · doi:10.5555/767345.767349 · Cited by 127
A Metamorphic Testing Approach for Online Testing of Service-Oriented Software Applications
Chan, W. K., Cheung, S.-C., & Leung, K. R. P. H. (2007) · International Journal of Web Services Research · doi:10.4018/jwsr.2007040103 · Cited by 127
Efficient unit test case minimization
Leitner, A., Oriol, M., Zeller, A., Ciupa, I., & Meyer, B. (2007) · doi:10.1145/1321631.1321698 · Cited by 121
Dynamic Analysis for Diagnosing Integration Faults
Mariani, L., Pastore, F., & Pezzè, M. (2010) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2010.93 · Cited by 121
Combined static and dynamic automated test generation
Zhang, S., Saff, D., Bu, Y., & Ernst, M. D. (2011) · Proceedings of the 2011 International Symposium on Software Testing and Analysis · doi:10.1145/2001420.2001463 · Cited by 118
Directed test suite augmentation: techniques and tradeoffs
Xu, Z., Kim, Y., Kim, M., Rothermel, G., & Cohen, M. B. (2010) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/1882291.1882330 · Cited by 118
Inferring Finite-State Models with Temporal Constraints
Walkinshaw, N., & Bogdanov, K. (2008) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2008.35 · Cited by 115
Modeling the Dynamics of UML State Machines
Börger, E., Cavarra, A., & Riccobene, E. (2000) · Lecture notes in computer science · doi:10.1007/3-540-44518-8_13 · Cited by 114
Wei, Y., Furia, C. A., Kazmin, N., & Meyer, B. (2011) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/1985793.1985820 · Cited by 111
Precise Documentation: The Key to Better Software
Parnas, D. L. (2010) · Springer eBooks · doi:10.1007/978-3-642-15187-3_8 · Cited by 108
Has the bug really been fixed?
Gu, Z., Barr, E. T., Hamilton, D. J., & Su, Z. (2010) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/1806799.1806812 · Cited by 107
Self-citationPrecise documentation of well-structured programs
Parnas, D. L., Madey, J., & Iglewski, M. (1994) · IEEE Transactions on Software Engineering · doi:10.1109/32.368133 · Cited by 106
Automatic detection of performance deviations in the load testing of Large Scale Systems
Malik, H., Hemmati, H., & Hassan, A. E. (2013) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2013.6606651 · Cited by 106
Evolving Readable String Test Inputs Using a Natural Language Model to Reduce Human Oracle Cost
Afshan, S., McMinn, P., & Stevenson, M. (2013) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2013.11 · Cited by 103
Self-citationAssessing Test Data Adequacy through Program Inference
Weyuker, E. J. (1983) · ACM Transactions on Programming Languages and Systems · doi:10.1145/69575.357231 · Cited by 102
Modelling Systems: Practical Tools and Techniques in Software Development
Fitzgerald, J. S., & Larsen, P. G. (2009) · Cambridge University Press eBooks · doi:10.1017/cbo9780511626975 · Cited by 102
A functional approach to program testing and analysis
Howden, W. E. (1986) · IEEE Transactions on Software Engineering · doi:10.1109/tse.1986.6313016 · Cited by 101
CrowdOracles: Can the Crowd Solve the Oracle Problem?
Pastore, F., Mariani, L., & Fraser, G. (2013) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2013.13 · Cited by 101
English as a formal specification language
Schwitter, R. (2004) · International Workshop on Database and Expert Systems Applications (DEXA) · doi:10.1109/dexa.2002.1045903 · Cited by 101
Pseudo-oracles for non-testable programs
Davis, M. D., & Weyuker, E. J. (1981) · Proceedings of the ACM '81 conference on - ACM 81 · doi:10.1145/800175.809889 · Cited by 100
High Level System Design and Analysis Using Abstract State Machines
Börger, E. (1999) · Lecture notes in computer science · doi:10.1007/3-540-48257-1_1 · Cited by 99
A subset of precise UML for model-based testing
Bouquet, F., Grandpierre, C., Legeard, B., Peureux, F., Vacelet, N., & Utting, M. (2007) · doi:10.1145/1291535.1291545 · Cited by 99
Assessing Oracle Quality with Checked Coverage
Schüler, D., & Zeller, A. (2011) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2011.32 · Cited by 97
Foundations of sequence-based software specification
Prowell, S., & Poore, J. H. (2003) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2003.1199071 · Cited by 97
ASM-Based Testing: Coverage Criteria and Automatic Test Sequence
Gargantini, A., & Riccobene, E. (2001) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.3217/jucs-007-11-1050 · Cited by 95
Programs, tests, and oracles: the foundations of testing revisited
Staats, M., Whalen, M. W., & Heimdahl, M. P. E. (2011) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/1985793.1985847 · Cited by 95
Metamorphic Testing and Its Applications
Zhou, Z. Q., Huang, D., Tse, T. H., Yang, Z., Huang, H., & Chen, T. Y. (2004) · Cited by 92
A comparative study of programmer-written and automatically inferred contracts
Polikarpova, N., Ciupa, I., & Meyer, B. (2009) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/1572272.1572284 · Cited by 92
Harman, M., Kim, S. G., Lakhotia, K., McMinn, P., & Yoo, S. (2010) · 2010 Third International Conference on Software Testing, Verification, and Validation Workshops · doi:10.1109/icstw.2010.31 · Cited by 85
Self-citationAutomated functional testing of online search services
Zhou, Z. Q., Zhang, S., Hagenbuchner, M., Tse, T. H., Kuo, F., & Chen, T. Y. (2010) · Software Testing Verification and Reliability · doi:10.1002/stvr.437 · Cited by 85
Automatic system testing of programs without test oracles
Murphy, C., Shen, K., & Kaiser, G. (2009) · Proceedings of the eighteenth international symposium on Software testing and analysis · doi:10.1145/1572272.1572295 · Cited by 84
Investigating the use of analysis contracts to improve the testability of object‐oriented code
Briand, L., Labiche, Y., & Sun, H. (2003) · Software Practice and Experience · doi:10.1002/spe.520 · Cited by 84
Finding Software Vulnerabilities by Smart Fuzzing
Bekrar, S., Bekrar, C., Groz, R., & Mounier, L. (2011) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2011.48 · Cited by 83
Testing against formal specifications: A theoretical view
Bernot, G. (1991) · Lecture notes in computer science · doi:10.1007/3540539816_63 · Cited by 80
Evolutionary algorithms for the multi‐objective test data generation problem
Ferrer, J., Chicano, F., & Alba, E. (2011) · Software Practice and Experience · doi:10.1002/spe.1135 · Cited by 80
A Comparative Study on Automated Software Test Oracle Methods
Shahamiri, S. R., Kadir, W. M. N. W., & Mohd-Hashim, S. Z. (2009) · International Conference on Software Engineering Advances · doi:10.1109/icsea.2009.29 · Cited by 78
An empirical investigation into branch coverage for C programs using CUTE and AUSTIN
Lakhotia, K., McMinn, P., & Harman, M. (2010) · Journal of Systems and Software · doi:10.1016/j.jss.2010.07.026 · Cited by 77
Self-citationGuderlei, R., & Mayer, J. (2007) · Asia-Pacific Conference on Quality Software (APAQS) · doi:10.1109/qsic.2007.4385527 · Cited by 76
Automatic detection of performance deviations in the load testing of large scale systems
Malik, H., Hemmati, H., & Hassan, A. E. (2013) · doi:10.5555/2486788.2486927 · Cited by 74
Automated metamorphic testing on the analyses of feature models
Segura, S., Hierons, R. M., Benavides, D., & Ruiz–Cortés, A. (2010) · Information and Software Technology · doi:10.1016/j.infsof.2010.11.002 · Cited by 71
Metamorphic Testing and Beyond
Chen, T. Y., Kuo, F., Tse, T. H., & Zhou, Z. Q. (2005) · International Workshop on Software Technology and Engineering Practice (STEP) · doi:10.1109/step.2003.18 · Cited by 71
Mouchawrab, S., Briand, L., Labiche, Y., & Penta, M. D. (2010) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2010.32 · Cited by 70
Requirements-based monitors for real-time systems
Peters, D. K., & Parnas, D. L. (2002) · IEEE Transactions on Software Engineering · doi:10.1109/32.988496 · Cited by 68
Staats, M., Gay, G., & Heimdahl, M. P. E. (2012) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2012.6227132 · Cited by 67
Murphy, C., Shen, K., & Kaiser, G. (2009) · 2009 International Conference on Software Testing Verification and Validation · doi:10.1109/icst.2009.19 · Cited by 65
Reducing qualitative human oracle costs associated with automatically generated test data
McMinn, P., Stevenson, M., & Harman, M. (2010) · doi:10.1145/1868048.1868049 · Cited by 64
Self-citationSpecification in B: An Introduction Using the B Toolkit
Lano, K., & Haughton, H. (1996) · IMPERIAL COLLEGE PRESS eBooks · doi:10.1142/p006 · Cited by 62
An automated framework for software test oracle
Shahamiri, S. R., Kadir, W. M. N. W., Ibrahim, S., & Hashim, S. Z. M. (2011) · Information and Software Technology · doi:10.1016/j.infsof.2011.02.006 · Cited by 62
Checking inside the black box: regression testing by comparing value spectra
Xie, T., & Notkin, D. S. (2005) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2005.107 · Cited by 62
Pezzè, M., & Zhang, C. (2014) · Advances in computers · doi:10.1016/b978-0-12-800160-8.00001-2 · Cited by 61
Software model synthesis using satisfiability solvers
Heule, M. J. H., & Verwer, S. (2012) · Empirical Software Engineering · doi:10.1007/s10664-012-9222-z · Cited by 60
Automated Session Data Repair for Web Application Regression Testing
Harman, M., & Alshahwan, N. (2008) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2008.56 · Cited by 60
Self-citationSearch-Based Test Input Generation for String Data Types Using the Results of Web Queries
McMinn, P., Shahbaz, M., & Stevenson, M. (2012) · 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation · doi:10.1109/icst.2012.94 · Cited by 59
Self-citationApplication of metamorphic testing in numerical analysis
Chan, F., Chen, T. Y., Cheung, S.-C., Lau, M., & Yiu, S. (1998) · Rare & Special e-Zone (The Hong Kong University of Science and Technology) · Cited by 59
Iterative Refinement of Reverse-Engineered Models by Model-Based Testing
Walkinshaw, N., Derrick, J., & Guo, Q. (2009) · Lecture notes in computer science · doi:10.1007/978-3-642-05089-3_20 · Cited by 59
Increasing Functional Coverage by Inductive Testing: A Case Study
Walkinshaw, N., Bogdanov, K., Derrick, J., & París, J. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-16573-3_10 · Cited by 58
Automated Software Testing Using Model-Checking
Sehgal, S. (2019) · Journal of Emerging Technologies and Innovative Research · Cited by 57
Daistish: systematic algebraic testing for OO programs in the presence of side-effects
Hughes, M., & Stotts, D. (1996) · ACM SIGSOFT Software Engineering Notes · doi:10.1145/226295.226301 · Cited by 56
Cause Reduction for Quick Testing
Groce, A., Alipour, M. A., Zhang, C., Chen, Y., & Regehr, J. D. (2014) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2014.37 · Cited by 55
Checking Memory Safety with Blast
Beyer, D., Henzinger, T. A., Jhala, R., & Majumdar, R. (2005) · Lecture notes in computer science · doi:10.1007/978-3-540-31984-9_2 · Cited by 54
Testing Algebraic Data Types and Processes: A Unifying Theory
Gaudel, M.-C., & James, P. R. (1998) · Formal Aspects of Computing · doi:10.1007/s001650050026 · Cited by 54
Formal derivation of strongly correct concurrent programs
van Lamsweerde, A., & Sintzoff, M. (1979) · Acta Informatica · doi:10.1007/bf00264015 · Cited by 54
Towards a Metamorphic Testing Methodology for Service-Oriented Software Applications
Chan, W. K., Cheung, S.-C., & Leung, K. R. P. H. (2006) · Asia-Pacific Conference on Quality Software (APAQS) · doi:10.1109/qsic.2005.67 · Cited by 54
Generating diverse software versions with genetic programming: an experimental study
Feldt, R. (1998) · IEE Proceedings - Software · doi:10.1049/ip-sen:19982444 · Cited by 54
Exploiting Common Object Usage in Test Case Generation
Fraser, G., & Zeller, A. (2011) · 2011 Fourth IEEE International Conference on Software Testing, Verification and Validation · doi:10.1109/icst.2011.53 · Cited by 53
Experiments with test case prioritization using relevant slices
Jeffrey, D., & Gupta, N. (2007) · Journal of Systems and Software · doi:10.1016/j.jss.2007.05.006 · Cited by 51
Automatically generating realistic test input from web services
Bozkurt, M., & Harman, M. (2011) · IEEE International Symposium on Service-Oriented System Engineering (SOSE) · doi:10.1109/sose.2011.6139088 · Cited by 50
Self-citationArtificial neural networks as multi-networks automated test oracle
Shahamiri, S. R., Kadir, W. M. N. W., Ibrahim, S., & Hashim, S. Z. M. (2011) · Automated Software Engineering · doi:10.1007/s10515-011-0094-z · Cited by 49
Cooper, S. B., & Leeuwen, J. V. (2013) · Elsevier eBooks · doi:10.1016/b978-0-12-386980-7.50019-8 · Cited by 49
Search-based failure discovery using testability transformations to generate pseudo-oracles
McMinn, P. (2009) · Genetic and Evolutionary Computation Conference (GECCO) · doi:10.1145/1569901.1570127 · Cited by 48
Self-citationFinding the needles in the haystack: Generating legal test inputs for object-oriented programs
Artzi, S., Ernst, M. D., Kieżun, A., Pacheco, C., & Perkins, J. H. (2006) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 46
Oracles for Distributed Testing
Hierons, R. M. (2012) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2011.45 · Cited by 42
Defining the meaning of tabular mathematical expressions
Jin, Y., & Parnas, D. L. (2010) · Science of Computer Programming · doi:10.1016/j.scico.2009.12.009 · Cited by 42
A Memory Allocation Profiler for C and Lisp Programs
Zorn, B. G., & Hilfinger, P. N. (1988) · UC Berkeley · Cited by 42
Understanding user understanding: determining correctness of generated program invariants
Staats, M., Hong, S., Kim, M., & Rothermel, G. (2012) · International Symposium on Software Testing and Analysis · doi:10.1145/2338965.2336776 · Cited by 40
Testing Java Components based on Algebraic Specifications
Yu, B., Kong, L., Zhang, Y., & Zhu, H. (2008) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2008.39 · Cited by 38
Simons, A. J. H. (2007) · Automated Software Engineering · doi:10.1007/s10515-007-0015-3 · Cited by 36
Document based rational software development
Parnas, D. L. (2008) · Knowledge-Based Systems · doi:10.1016/j.knosys.2008.11.001 · Cited by 36
Detecting Communication Protocol Security Flaws by Formal Fuzz Testing and Machine Learning
Shu, G., Hsu, Y., & Lee, D. (2008) · Lecture notes in computer science · doi:10.1007/978-3-540-68855-6_19 · Cited by 35
Formal specifications and test: Correctness and oracle
Gall, P. L., & Arnould, A. (1996) · Lecture notes in computer science · doi:10.1007/3-540-61629-2_52 · Cited by 35
Integration Testing of Components Guided by Incremental State Machine Learning
Ke-qin, L., Groz, R., & Shahbaz, M. (2006) · Testing: Academic and Industrial Conference Practice and Research Techniques (TAIC PART) · doi:10.1109/taic-part.2006.15 · Cited by 35
Self-citationGarland, S. J., Guttag, J. V., & Horning, J. J. (1993) · Lecture notes in computer science · doi:10.1007/3-540-56883-2_15 · Cited by 34
Modular System Verification by Inference, Testing and Reachability Analysis
Groz, R., Ke-qin, L., Petrenko, A., & Shahbaz, M. (2008) · Lecture notes in computer science · doi:10.1007/978-3-540-68524-1_16 · Cited by 34
Self-citationLearning Component Interfaces with May and Must Abstractions
Singh, R., Giannakopoulou, D., & Păsăreanu, C. S. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-14295-6_45 · Cited by 33
A note on test oracles and semantics of algebraic specifications
Zhu, H. (2003) · Asia-Pacific Conference on Quality Software (APAQS) · doi:10.1109/qsic.2003.1319090 · Cited by 33
Context- and path-sensitive memory leak detection
Xie, Y., & Aiken, A. (2005) · ACM SIGSOFT Software Engineering Notes · doi:10.1145/1095430.1081728 · Cited by 33
Elkind, E., Genest, B., Peled, D., & Hongyang Qu, B. (2006) · Lecture notes in computer science · doi:10.1007/11888116_30 · Cited by 32
Automated Black-Box Testing with Abstract VDM Oracle
Aichernig, B. K. (1999) · Lecture notes in computer science · doi:10.1007/3-540-48249-0_22 · Cited by 31
Automating image segmentation verification and validation by learning test oracles
Frounchi, K., Briand, L. C., Grady, L., Labiche, Y., & Subramanyan, R. (2011) · Information and Software Technology · doi:10.1016/j.infsof.2011.06.009 · Cited by 31
On the Use of Specification-Based Assertions as Test Oracles
Coppit, D. W., & Haddox-Schatz, J. M. (2006) · IEEE Software Engineering Workshop (SEW) · doi:10.1109/sew.2005.33 · Cited by 31
An Empirical Evaluation of Assertions as Oracles
Shrestha, K., & Rutherford, M. J. (2011) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2011.50 · Cited by 30
Oracle-Centric Test Case Prioritization
Staats, M., Loyola, P., & Rothermel, G. (2012) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre.2012.13 · Cited by 29
Metamorphic Testing of Stochastic Optimisation
Yoo, S. (2010) · IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) · doi:10.1109/icstw.2010.26 · Cited by 29
Self-citationTest oracles using statistical methods
Mayer, J., & Guderlei, R. (2004) · Cited by 28
Dodona: automated oracle data set selection
Loyola, P., Staats, M., Ko, I., & Rothermel, G. (2014) · International Symposium on Software Testing and Analysis · doi:10.1145/2610384.2610408 · Cited by 28
Detecting Anomaly and Failure in Web Applications
Ricca, F., & Tonella, P. (2006) · IEEE Multimedia · doi:10.1109/mmul.2006.26 · Cited by 27
On Oracles for Interpreting Test Results against Algebraic Specifications
Machado, P. D. L. (1998) · Lecture notes in computer science · doi:10.1007/3-540-49253-4_35 · Cited by 26
Stateful testing: Finding more errors in code and contracts
Wei, Y., Roth, H., Furia, C. A., Pei, Y., Horton, A., Steindorfer, M. J., Nordio, M., & Meyer, B. (2011) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2011.6100094 · Cited by 25
Using transient/persistent errors to develop automated test oracles for event-driven software
Memon, A. M., & Xie, Q. (2004) · Automated Software Engineering · doi:10.1109/ase.2004.73 · Cited by 24
Verdict functions in testing with a fault domain or test hypotheses
Hierons, R. M. (2009) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/1538942.1538944 · Cited by 23
Deadlocks and livelocks in transition systems
Sifakis, J. (2005) · Springer eBooks · doi:10.1007/bfb0022535 · Cited by 23
LOFT: A tool for assisting selection of test data sets from algebraic specifications
Marre, B. (1995) · Lecture notes in computer science · doi:10.1007/3-540-59293-8_240 · Cited by 22
Searching for invariants using genetic programming and mutation testing
Ratcliff, S., White, D. R., & Clark, J. A. (2011) · Genetic and Evolutionary Computation Conference (GECCO) · doi:10.1145/2001576.2001832 · Cited by 22
Abstraction in Assertion-Based Test Oracles
Cheon, Y. (2007) · Asia-Pacific Conference on Quality Software (APAQS) · doi:10.1109/qsic.2007.4385528 · Cited by 21
Enabling the runtime assertion checking of concurrent contracts for the Java modeling language
Araujo, W., Briand, L., & Labiche, Y. (2011) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/1985793.1985903 · Cited by 21
A Comparison of Tabular Expression-Based Testing Strategies
Feng, X., Parnas, D. L., Tse, T. H., & O'Callaghan, T. (2011) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2011.78 · Cited by 21
Using transient/persistent errors to develop automated test oracles for event-driven software
Memon, A. M., & Xie, Q. (2004) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2004.1342736 · Cited by 21
Managing Variability for Software Product Lines: Working with Variability Mechanisms
Clements, P. (2006) · International Software Product Line Conference (SPLC) · doi:10.1109/spline.2006.1691594 · Cited by 20
Automated Inference of Models for Black Box Systems Based on Interface Descriptions
Merten, M., Howar, F. M., Steffen, B., Pelliccione, P., & Tivoli, M. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-34026-0_7 · Cited by 18
AMC: An Adaptive Model Checker
Groce, A., Peled, D., & Yannakakis, M. (2002) · Lecture notes in computer science · doi:10.1007/3-540-45657-0_44 · Cited by 18
Proving Total Correctness of Parallel Programs
Babich, A. F. (1979) · IEEE Transactions on Software Engineering · doi:10.1109/tse.1979.230192 · Cited by 18
Research works printed but not linked: 6
- A. Avizienis and L. Chen, "On the implementation of N-version programming for software fault-tolerance during execution," in Proc. 1st Int. Comput. Softw. Appl. Conf., 1977, pp. 149-155.
- P. McMinn, M. Stevenson, and M. Harman, "Reducing qualita- tive human oracle costs associated with automatically generated test data," in Proc. Int. Workshop Softw. Test Output Validation, Jul. 2010, pp. 1-4. · DOI
- A. J. Offutt, J. Pan, and J. M. Voas, "Procedures for reducing the size of coverage-based test sets," in Int. Conf. Testing Comput. Softw., 1995, pp. 111-123.
- M. Shahbaz, P. McMinn, and M. Stevenson, "Automatic genera- tion of valid and invalid test data for string validation routines," Sci. Comput. Programm., vol. 97, no. 4, pp. 405-425, 2015.
- J. M. Spivey, Z Notation-a Reference Manual, 2nd ed. Englewood Cliffs, NJ, USA: Prentice-Hall, 1992.
- G. V. Bochmann, C. He, and D. Ouimet, "Protocol testing using automatic trace analysis," in Proc. Can. Conf. Elect. Comput. Eng., 1989, pp. 814-820.
Cited by
Machine Learning Testing: Survey, Landscapes and Horizons
Zhang, J. M., Harman, M., Ma, L., & Liu, Y. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2962027 · Cited by 897
Self-citationMetamorphic Testing: A Review of Challenges and Opportunities
Chen, T. Y., Kuo, F.-C., Liu, H., Poon, P.-L., Towey, D., Tse, T. H., & Zhou, Z. Q. (2019) · ACM Computing Surveys · doi:10.1145/3143561 · Cited by 665
Large language models for software engineering: survey and open problems
Fan, A., Gokkaya, B., Harman, M., Lyubarskiy, M., Sengupta, S., Yoo, S., & Zhang, J. M. (2023) · 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE) · doi:10.1109/icse-fose59343.2023.00008 · Cited by 650
“3) Test Correctness: Traditional software test generation has suffered from the Oracle Problem [6], i.e., they are inhibited by the lack of an automated oracle that determines whether a test outcome is correct.”
BackgroundMethodSelf-citationTrustworthy AI: From Principles to Practices
Li, B., Qi, P., Liu, B., Di, S., Liu, J., Pei, J., Yi, J., & Zhou, B. (2023) · ACM Computing Surveys · doi:10.1145/3555803 · Cited by 612
“Similar to canonical software testing, the problem to automatically generate the expected ground truth, known as the oracle problem [34], also occurs in the AI software testing scenario.”
MethodSoftware Testing With Large Language Models: Survey, Landscape, and Vision
Wang, J., Huang, Y., Chen, C., Liu, Z., Wang, S., & Wang, Q. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3368208 · Cited by 573
“A test oracle is a source of information about whether the output of a software system (or program or function or method) is correct or not [136].”
Metadata
- Authors
- Barr, E. T., Harman, M., McMinn, P., Shahbaz, M., & Yoo, S.
- Affiliations
- University College London (GB)
- University of Sheffield (GB)
- Venue
- IEEE Transactions on Software Engineering
- Published
- 2015-05-01
- Volume
- 41
- Issue
- 5
- Pages
- 507-525
- ISSN
- 0098-5589 (ISSN-L)
- 1939-3520
- OpenAlex
- W2041713059
- Cited by
- 1,104
- References
- 236
- Citations by year
- 2026: 123 · 2025: 132 · 2024: 116 · 2023: 96 · 2022: 104 · 2021: 114 · 2020: 102 · 2019: 89 · 2018: 98 · 2017: 56 · 2016: 58 · 2015: 16
- FWCI
- 151.12
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- IEEE Transactions on Software Engineering (publishedVersion, cc-by)
- UCL Discovery (University College London) (submittedVersion)
- Funders
- Engineering and Physical Sciences Research Council (EP/I010165/1, EP/I010386/1)
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Software › Advanced Software Engineering Methodologies
- Keywords
- oracle problem, software testing, metamorphic testing, informal specification, test automation, fault detection
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.