The DOI matches its Crossref record; 218 of the 236 references it cites were found in a registry or an index, and 2 are not checked yet.
The Oracle Problem in Software Testing: A Survey
Open accessAbstract
Testing involves examining the behaviour of a system in order to discover potential faults. Given an input for a system, the challenge of distinguishing the corresponding desired, correct behaviour from potentially incorrect behavior is called the “test oracle problem”. Test oracle automation is important to remove a current bottleneck that inhibits greater overall test automation. Without test oracle automation, the human has to determine whether observed behaviour is correct. The literature on test oracles has introduced techniques for oracle automation, including modelling, specifications, contract-driven development and metamorphic testing. When none of these is completely adequate, the final source of test oracle information remains the human, who may be aware of informal specifications, expectations, norms and domain specific information that provide informal oracle guidance. All forms of test oracles, even the humble human, involve challenges of reducing cost and increasing benefit. This paper provides a comprehensive survey of current approaches to the test oracle problem and an analysis of trends in this important area of software testing research and practice.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The metric depends on rigorous testing, which is currently limited by the lack of test oracles [11].”
From A 2030 Roadmap for Software Engineering (2025) · cites this paper as [11]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Statecharts: a visual formalism for complex systems
Harel, D. (1987) · Science of Computer Programming · doi:10.1016/0167-6423(87)90035-9 · Cited by 6,751
An Axiomatic Basis for Computer Programming
Hoare, T. (1968) · Cited by 4,335
An axiomatic basis for computer programming
Hoare, C. A. R. (1969) · Communications of the ACM · doi:10.1145/363235.363259 · Cited by 4,002
Holzmann, G. J. (1997) · IEEE Transactions on Software Engineering · doi:10.1109/32.588521 · Cited by 3,803
Metaheuristics in combinatorial optimization: Overview and conceptual comparison
Blum, C., & Roli, A. (2003) · ACM Computing Surveys · doi:10.1145/937503.937505 · Cited by 3,236
Symbolic execution and program testing
King, J. C. (1976) · Communications of the ACM · doi:10.1145/360248.360252 · Cited by 3,031
Baader, F., & Nipkow, T. (1998) · Cambridge University Press eBooks · doi:10.1017/cbo9781139172752 · Cited by 2,466
Metaheuristics in Combinatorial Optimization
Gendreau, M., & Potvin, J.-Y. (2005) · Annals of Operations Research · doi:10.1007/s10479-005-3971-7 · Cited by 2,338
Learning regular sets from queries and counterexamples
Angluin, D. (1987) · Information and Computation · doi:10.1016/0890-5401(87)90052-6 · Cited by 2,114
DART: directed automated random testing
Godefroid, P., Klarlund, N., & Sen, K. (2005) · ACM SIGPLAN Notices · doi:10.1145/1064978.1065036 · Cited by 2,067
Research works printed but not linked: 7
- Avizienis, 1977, On the implementation of N-version programming for software fault-tolerance during execution,, 149
- 10.4018/978-1-60566-060-8.ch170 · doi:10.4018/978-1-60566-060-8.ch170
- 10.1007/978-1-4471-3435-0 · doi:10.1007/978-1-4471-3435-0
- 10.1145/1868048.1868049 · DOI
- Offutt, 1995, Procedures for reducing the size of coverage-based test sets,, Int. Conf. Testing Comput. Softw., 111
- Spivey, 1992, Z Notationa Reference Manual
- Bochmann, 1989, Protocol testing using automatic trace analysis, 814
Cited by
Machine Learning Testing: Survey, Landscapes and Horizons
Zhang, J. M., Harman, M., Ma, L., & Liu, Y. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2962027 · Cited by 897
Self-citationMetamorphic Testing: A Review of Challenges and Opportunities
Chen, T. Y., Kuo, F.-C., Liu, H., Poon, P.-L., Towey, D., Tse, T. H., & Zhou, Z. Q. (2019) · ACM Computing Surveys · doi:10.1145/3143561 · Cited by 665
Large language models for software engineering: survey and open problems
Fan, A., Gokkaya, B., Harman, M., Lyubarskiy, M., Sengupta, S., Yoo, S., & Zhang, J. M. (2023) · 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE) · doi:10.1109/icse-fose59343.2023.00008 · Cited by 650
“3) Test Correctness: Traditional software test generation has suffered from the Oracle Problem [6], i.e., they are inhibited by the lack of an automated oracle that determines whether a test outcome is correct.”
BackgroundMethodSelf-citationTrustworthy AI: From Principles to Practices
Li, B., Qi, P., Liu, B., Di, S., Liu, J., Pei, J., Yi, J., & Zhou, B. (2023) · ACM Computing Surveys · doi:10.1145/3555803 · Cited by 612
“Similar to canonical software testing, the problem to automatically generate the expected ground truth, known as the oracle problem [34], also occurs in the AI software testing scenario.”
MethodSoftware Testing With Large Language Models: Survey, Landscape, and Vision
Wang, J., Huang, Y., Chen, C., Liu, Z., Wang, S., & Wang, Q. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3368208 · Cited by 573
Metadata
- Authors
- Barr, E. T., Harman, M., McMinn, P., Shahbaz, M., & Yoo, S.
- Affiliations
- University College London (GB)
- University of Sheffield (GB)
- Venue
- IEEE Transactions on Software Engineering
- Published
- 2015-05-01
- Volume
- 41
- Issue
- 5
- Pages
- 507-525
- ISSN
- 0098-5589 (ISSN-L)
- 1939-3520
- OpenAlex
- W2041713059
- Cited by
- 1,104
- References
- 236
- Citations by year
- 2026: 123 · 2025: 132 · 2024: 116 · 2023: 96 · 2022: 104 · 2021: 114 · 2020: 102 · 2019: 89 · 2018: 98 · 2017: 56 · 2016: 58 · 2015: 16
- FWCI
- 151.12
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- IEEE Transactions on Software Engineering (publishedVersion, cc-by)
- UCL Discovery (University College London) (submittedVersion)
- Funders
- Engineering and Physical Sciences Research Council (EP/I010165/1, EP/I010386/1)
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Software › Advanced Software Engineering Methodologies
- Keywords
- oracle problem, software testing, metamorphic testing, informal specification, test automation, fault detection
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: The Oracle Problem in Software Testing: A Survey. https://app.cochat.ai/papers/10.1109/tse.2014.2372785
- The paper
- Barr, E. T., Harman, M., McMinn, P., Shahbaz, M., & Yoo, S. (2015). The Oracle Problem in Software Testing: A Survey. IEEE Transactions on Software Engineering. https://doi.org/10.1109/tse.2014.2372785
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.