The Oracle Problem in Software Testing: A Survey
Open accessThe DOI matches its Crossref record; 218 of the 236 references it cites were found in a registry or an index.
Abstract
Testing involves examining the behaviour of a system in order to discover potential faults. Given an input for a system, the challenge of distinguishing the corresponding desired, correct behaviour from potentially incorrect behavior is called the “test oracle problem”. Test oracle automation is important to remove a current bottleneck that inhibits greater overall test automation. Without test oracle automation, the human has to determine whether observed behaviour is correct. The literature on test oracles has introduced techniques for oracle automation, including modelling, specifications, contract-driven development and metamorphic testing. When none of these is completely adequate, the final source of test oracle information remains the human, who may be aware of informal specifications, expectations, norms and domain specific information that provide informal oracle guidance. All forms of test oracles, even the humble human, involve challenges of reducing cost and increasing benefit. This paper provides a comprehensive survey of current approaches to the test oracle problem and an analysis of trends in this important area of software testing research and practice.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures











Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The metric depends on rigorous testing, which is currently limited by the lack of test oracles [11].”
From A 2030 Roadmap for Software Engineering (2025) · cites this paper as [11]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Items 201–220 of 220
Testing from Formal Specifications, a Generic Approach
Gaudel, M.-C. (2001) · Lecture notes in computer science · doi:10.1007/3-540-45136-6_3 · Cited by 17
Mining test oracles of web search engines
Zheng, W., Ma, H., Lyu, M. R.-T., Xie, T., & King, I. (2011) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2011.6100085 · Cited by 17
Araujo, W., Briand, L., & Labiche, Y. (2011) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1109/esem.2011.9 · Cited by 17
Using information about functions in selecting test cases
Clermont, M., & Parnas, D. L. (2005) · ACM SIGSOFT Software Engineering Notes · doi:10.1145/1082983.1083276 · Cited by 17
Using Behaviour Inference to Optimise Regression Test Sets
Taylor, R. G., Hall, M., Bogdanov, K., & Derrick, J. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-34691-0_14 · Cited by 16
The detection of dangling references in C++ programs
Eyre-Todd, R. A. (1993) · ACM Letters on Programming Languages and Systems · doi:10.1145/176454.176504 · Cited by 16
Improving the accuracy of oracle verdicts through automated model steering
Gay, G., Rayadurgam, S., & Heimdahl, M. P. E. (2014) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/2642937.2642989 · Cited by 15
An approach to testing specifications
Jard, C., & von Bochmann, G. (1983) · Journal of Systems and Software · doi:10.1016/0164-1212(83)90018-3 · Cited by 14
Simulating the behaviour of software modules by trace rewriting
Wang, Y., & Parnas, D. L. (1993) · doi:10.5555/257572.257577 · Cited by 13
Liberating the programmer with prorogued programming
Afshari, M., Barr, E. T., & Su, Z. (2012) · ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software (Onward!) · doi:10.1145/2384592.2384595 · Cited by 13
Self-citationSimulating the behaviour of software modules by trace rewriting
Wang, Y., & Parnas, D. L. (2002) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.1993.346059 · Cited by 12
Towards Reconciling Use Cases via Controlled Language and Graphical Models
Böttger, K., Schwitter, R., Mollá, D., & Richards, D. (2003) · Lecture notes in computer science · doi:10.1007/3-540-36524-9_10 · Cited by 9
Using information about functions in selecting test cases
Clermont, M., & Parnas, D. L. (2005) · doi:10.1145/1083274.1083276 · Cited by 9
Wang, Y., & Parnas, D. L. (1993) · Lecture notes in computer science · doi:10.1007/3-540-56393-8_26 · Cited by 8
Automated Session Data Repair for Web Application Regression Testing
Alshahwan, N., & Harman, M. (2008) · UCL Discovery (University College London) · Cited by 8
Self-citationCorrect Systems: Building a Business Process Solution
(2000) · Kybernetes · doi:10.1108/k.2000.29.1.144.5 · Cited by 7
Tabular Expression-Based Testing Strategies: A Comparison
Feng, X., Parnas, D. L., & Tse, T. H. (2007) · Testing: Academic and Industrial Conference Practice and Research Techniques (TAIC PART) · doi:10.1109/taic.part.2007.19 · Cited by 7
Proving programs by sets of computations
Blikle, A. (1975) · Lecture notes in computer science · doi:10.1007/3-540-07162-8_694 · Cited by 7
Automated support for testing and debugging of real-time programs using oracles
Dillon, L. K. (2000) · ACM SIGSOFT Software Engineering Notes · doi:10.1145/340855.340884 · Cited by 3
Metamorphic testing and its applications
Chen, T. Y., Huang, D. H., Huang, H., Tse, T.-H., Yang, Z. Y., & Quan, Z. Z. (2004) · Swinburne figshare (Swinburne University of Technology) · Cited by 2
Research works printed but not linked: 6
- A. Avizienis and L. Chen, "On the implementation of N-version programming for software fault-tolerance during execution," in Proc. 1st Int. Comput. Softw. Appl. Conf., 1977, pp. 149-155.
- P. McMinn, M. Stevenson, and M. Harman, "Reducing qualita- tive human oracle costs associated with automatically generated test data," in Proc. Int. Workshop Softw. Test Output Validation, Jul. 2010, pp. 1-4. · DOI
- A. J. Offutt, J. Pan, and J. M. Voas, "Procedures for reducing the size of coverage-based test sets," in Int. Conf. Testing Comput. Softw., 1995, pp. 111-123.
- M. Shahbaz, P. McMinn, and M. Stevenson, "Automatic genera- tion of valid and invalid test data for string validation routines," Sci. Comput. Programm., vol. 97, no. 4, pp. 405-425, 2015.
- J. M. Spivey, Z Notation-a Reference Manual, 2nd ed. Englewood Cliffs, NJ, USA: Prentice-Hall, 1992.
- G. V. Bochmann, C. He, and D. Ouimet, "Protocol testing using automatic trace analysis," in Proc. Can. Conf. Elect. Comput. Eng., 1989, pp. 814-820.
Cited by
Machine Learning Testing: Survey, Landscapes and Horizons
Zhang, J. M., Harman, M., Ma, L., & Liu, Y. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2962027 · Cited by 897
Self-citationMetamorphic Testing: A Review of Challenges and Opportunities
Chen, T. Y., Kuo, F.-C., Liu, H., Poon, P.-L., Towey, D., Tse, T. H., & Zhou, Z. Q. (2019) · ACM Computing Surveys · doi:10.1145/3143561 · Cited by 665
Large language models for software engineering: survey and open problems
Fan, A., Gokkaya, B., Harman, M., Lyubarskiy, M., Sengupta, S., Yoo, S., & Zhang, J. M. (2023) · 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE) · doi:10.1109/icse-fose59343.2023.00008 · Cited by 650
“3) Test Correctness: Traditional software test generation has suffered from the Oracle Problem [6], i.e., they are inhibited by the lack of an automated oracle that determines whether a test outcome is correct.”
BackgroundMethodSelf-citationTrustworthy AI: From Principles to Practices
Li, B., Qi, P., Liu, B., Di, S., Liu, J., Pei, J., Yi, J., & Zhou, B. (2023) · ACM Computing Surveys · doi:10.1145/3555803 · Cited by 612
“Similar to canonical software testing, the problem to automatically generate the expected ground truth, known as the oracle problem [34], also occurs in the AI software testing scenario.”
MethodSoftware Testing With Large Language Models: Survey, Landscape, and Vision
Wang, J., Huang, Y., Chen, C., Liu, Z., Wang, S., & Wang, Q. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3368208 · Cited by 573
“A test oracle is a source of information about whether the output of a software system (or program or function or method) is correct or not [136].”
Metadata
- Authors
- Barr, E. T., Harman, M., McMinn, P., Shahbaz, M., & Yoo, S.
- Affiliations
- University College London (GB)
- University of Sheffield (GB)
- Venue
- IEEE Transactions on Software Engineering
- Published
- 2015-05-01
- Volume
- 41
- Issue
- 5
- Pages
- 507-525
- ISSN
- 0098-5589 (ISSN-L)
- 1939-3520
- OpenAlex
- W2041713059
- Cited by
- 1,104
- References
- 236
- Citations by year
- 2026: 123 · 2025: 132 · 2024: 116 · 2023: 96 · 2022: 104 · 2021: 114 · 2020: 102 · 2019: 89 · 2018: 98 · 2017: 56 · 2016: 58 · 2015: 16
- FWCI
- 151.12
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- IEEE Transactions on Software Engineering (publishedVersion, cc-by)
- UCL Discovery (University College London) (submittedVersion)
- Funders
- Engineering and Physical Sciences Research Council (EP/I010165/1, EP/I010386/1)
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Software › Advanced Software Engineering Methodologies
- Keywords
- oracle problem, software testing, metamorphic testing, informal specification, test automation, fault detection
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.